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Executive Summary of Findings: 
Year 5 Implementation and Impact 


A. Project Overview 


This report summarizes the results of the Newark, NJ, Striving Readers program for project Years 1, 
2, 3, 4, and 5. This report updates one analysis (3 years); the remainder of the impacts and 
implementation findings are for Year 4 and have been reported previously in Saiving Readers Study: 
Targeted ¢ Whole-School Interventions — Year 4. The Striving Readers Grant addresses the unmet needs 
of middle school students reading 2 or more years below grade level and provides professional 
development for teachers in all core content areas to help them learn about and use mote effective 
literacy strategies. Nineteen middle schools in Newark are participating in the U.S. Department of 
Education Striving Readers study. Two components of the project are being evaluated: a targeted 


intervention and a whole-school intervention. 


B. Targeted Intervention 


Scholastic’s READ 180 Enterprise Edition was chosen to be the targeted intervention and replaced 
the core language arts curriculum for targeted students in the treatment schools. READ 180 directly 
addresses the individual needs of adolescents reading below grade level by using adaptive and 
instructional software, high-interest literature, and direct instruction. Teachers received training on 
all aspects of the READ 180 curriculum, from preparation to implementation and evaluation. In 
addition, teachers received training on using student data for differentiated instruction and 


instruction on interpreting READ 180 data reports. 


B.1 Description of Schools and Students in Targeted Intervention 


The schools eligible to participate in the Striving Readers program were randomly assigned to either 
the treatment or a control condition in May 2006. No classroom- or student-level random 


assignment was involved. Eligible middle schools were identified based on the following criteria: 


a Be Title I eligible 


: Serve a minimum of two grades (from 6, 7, 8) 
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. Not already using READ 180 
a Be categorized as “in need of improvement” under the No Child Left Behind Act 


a Serve a minimum of 25 eligible students 


These criteria ultimately resulted in a pool of 19 schools for randomization. Ten schools were 


randomly assigned to the treatment group. 


Students were identified as eligible based on their score on the reading subtest of the New Jersey 
Assessment of Skills and Knowledge (NJASK). In Year 4, 1,070 students participated in the 
intervention. Of those, 577 attended treatment schools and 493 attended control schools. Table A 


shows the distribution of these students by select demographics and by treatment group. 


Table A. Characteristics of students in the targeted intervention in Year 4 


Students in Students in All targeted 
Number (column %) treatment schools control schools students 
Total number of students 577 493 1,070 
(54%) (46%) 

Average no. of students per school 57.7 54.8 56.3 
Grade 

6th grade 35% 38% 36% 

7th grade 34%* 32% 33% 

8th grade 31%+ 30% 30% 
Gender 

Male 60% 63% 61% 

Female 40% 37% 39% 
Status 

Economically disadvantaged 62% 61% 62% 

English language learners (ELLs) 15% 11% 13% 

Special education 46% 42% 44% 
Race/ethnicity 

African American 53% 57% 55% 

Hispanic 43% 41% 42% 

Caucasian 1% 1% 1% 

Other 2% 1% 1% 


* Two years of exposure. 
+Three years of exposure. 


= The economic status was not provided for 322 students (178 treatment students and 144 control students). 
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B.2 Summary of the Targeted Intervention Implementation Findings 


To determine the degree of fidelity to READ 180, multiple components were evaluated for each 
READ 180 teacher. These components are training, class size, ongoing student assessments, and 
instructional software. An overall implementation summary score was developed for each school in 
Year 4. Table B provides each school’s score for the multiple components of the targeted 


intervention and an overall implementation score. 


Table B. Average school-level summary scores for implementation in Year 4 


Ongoing Summary 
Treatment Professional student Instructional Average implementation 
school development* Class size assessment software use score scores 

School 10 - 4 4 3 3.7 High 
School 5 - 4 4 2.5 3.5 High 
School 4 - 4 4 1.5 3.2 High 
School 17 - 4 4 1.5 3.2 High 
School 6 - 4 4 3.0 Moderate-to-High 
School 8 - 4 4 1 3.0 Moderate-to-High 
School 13 - 4 4 1 3.0 Moderate-to-High 
School 14 - 4 4 1 3.0 Moderate-to-High 
School 15 - 4 4 1 3.0 Moderate-to-High 
School 16 - 4 4 1 3.0 Moderate-to-High 


* A summary score was not calculated for professional development due to a change in the professional development model. 


Forty percent of schools achieved high implementation in all three components. The remaining 60 
percent schools had moderate-to-high implementation. All of the schools fully implemented 
ongoing monitoring of student progress in reading comprehension and all were within READ 180 


guidelines for class size. 


B.3 Summary of the Targeted Intervention Impact Findings 


Based on analyses from 5 years of Striving Readers data, READ 180 had an overall significant effect 
in one area of literacy skill. After 2 years of exposure to READ 180, a significant effect in reading 
comprehension was observed for students in the treatment group. However, students with 1 year or 


3 years of exposure exhibited the same level of achievement as students in control schools. These 


findings suggest that struggling adolescent readers who are 2 or more years below grade level in 
reading may need 2 years of exposure to the intervention before significant impacts can be observed. 


One year may not provide a high enough dose, and 3 years may be too much. 


Significant effects were also found for subgroups of students, as shown in Table C. READ 180 
appeared to be particularly effective for special education students. Special education students with 1 
year of exposure to READ 180 scored significantly higher on the Vocabulary subtest than control 
students. Special education students with 2 years of treatment scored higher than control students 
on the Comprehension subtest. However, this improvement was not sustained for special education 
students with 3 years of exposure to READ 180. On the other hand, these same students had better 
school attendance compated to special education students in the control group with 3 years of 


exposure. 


Table C. Summary of analysis findings by subgroups 


African Special 

Analysis Overall Female Male American Hispanic Education 
groups Outcomes ES Sig ES Sig ES’ Sig ES Sig ES Sig ES’ Sig 
1. 1 year Attendance 
of exposure 

Vocabulary v 

Comprehension 

Language Arts 
2.2years Attendance 
of exposure 

Vocabulary 

Comprehension v v v v v 

Language Arts 
3.3 years Attendance v v 
of exposure 

Vocabulary 


Comprehension 


Language Arts 


Y Positive at either p<.05 or effect size>.20 


Another subgroup that appeared to benefit from READ 180 was males. Male students with 2 years 
of exposure scored significantly higher on the Comprehension subtest; this significant finding also 
had substantial effect sizes (0.21). However, male students with 3 years of exposure did not 
demonstrate continued improvement in their reading comprehension. Male students with 3 years of 


exposure had fewer absent days than those students in the control groups. 
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African American students also showed positive effects in Comprehension after 2 years. This effect 
was not sustained after 3 years of exposure. Among Hispanic students, few significant effects were 

observed. Hispanic students with 2 years of READ 180 exposure scored significantly higher on the 
Language Arts subtest than the control group. None of the other findings were either statistically or 


practically significant, including effects after 3 years of exposure to the program. 


C. Whole-School Intervention 


The goal of Newark Public Schools’ (NPS) whole-school intervention is to improve students’ ability 
to “read to learn” across multiple content areas. The whole-school intervention is thus designed to 
train teachers to better integrate different learning strategies within the district’s core literacy 
program for middle-grade students. To this end, the intervention provides professional development 
to bolster the literacy knowledge of teachers of grades 6, 7, and 8 in large-group settings and to 
provide direct coaching support during in-school visits. These professional development and 
support activities are conducted by experts from New Jersey City University (NJCU) and the 
National Urban Alliance (NUA). Using a train-the-trainers model, the Resource Teacher 
Coordintors (RTCs) support the implementation of both professional development approaches 


through their own large-group training and site-based demonstration lessons and coaching. 


C.1 Description of Schools and Teachers in Whole-School Intervention 


The 19 schools participating in the targeted intervention are the same as those participating in the 
whole-school intervention. However, the whole-school intervention is not being evaluated with a 
randomized design, and so all eligible teachers in all 19 schools receive the intervention. 

In Year 4, 337 teachers were eligible to recetve professional development as part of the whole- 
school intervention. Of these, 138 were eligible for professional development provided by the NUA 
(teachers who taught on/y math, science, or social studies). Fifty-nine’ other teachers were eligible for 
training from NJCU (teachers who taught ov/y language arts). In addition, 140 teachers were eligible 
for both NUA and NJCU training. These teachers either taught both language arts and a content 
area subject (usually social studies), or they taught all subjects (usually special education or bilingual 


teachets). 


' Includes 19 literacy coaches. 
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Students in all 19 Striving Readers schools, across the 6th, 7th, and 8th grades, were exposed to the 


whole-school intervention. 


C.2 Summary of the Whole-School Intervention Implementation 
Findings 


The summary scale was developed in Year 2 to describe how connected professional development 
inputs are involved in the whole-school intervention model was used again for Year 4. Table D 
provides each school’s score for the multiple components of the whole-school intervention 
professional development—the group training sessions and the in-school coaching visits—for the 
NUA and the NJCU intervention models. In addition, an overall implementation score and level of 


implementation are calculated for each school in the study. 


Table D. School-level summary scores for participation in whole-school intervention in Year 4 


Implementation scores by component 


NUA NJCU Summary 
Large-group In-school Large-group In-school Average implementation 
School training coaching training coaching score scores 
School 1 1 4 3 4 3 Moderate-to-High 
School 2 1 4 3 4 3 Moderate-to-High 
School 3 3 4 2 4 3.25 Moderate-to-High 
School 5 2 4 3 4 3.25 Moderate-to-High 
School 6 2 4 2 4 3 Moderate-to-High 
School 7 3 4 2 4 3.25 Moderate-to-High 
School 8 2 4 2 4 3 Moderate-to-High 
School 9 1 4 3 4 3 Moderate-to-High 
School 10 2 4 2 4 3 Moderate-to-High 
School 12 3 4 2 4 3.25 Moderate-to-High 
School 13 2 4 2 4 3 Moderate-to-High 
School 14 3 4 2 4 3.25 Moderate-to-High 
School 15 3 4 1 4 3 Moderate-to-High 
School 18 3 4 2 4 3.25 Moderate-to-High 
School 19 3 4 1 4 3 Moderate-to-High 
School 4 1 4 2 4 2.75 Moderate 
School 11 1 4 1 4 2.5 Moderate 
School 16 2 4 1 4 2.75 Moderate 
School 17 1 4 1 4 2.5 Moderate 
“Average =——sia2005'ti(‘<isé«dKz«zsC“(<(i‘sé‘iSS©6©6CU40002~*é—“‘“‘«CSSt~SC*é«~ Moderate to High 


As Table D shows, no school achieved full implementation of all four components. However, 15 


schools had moderate-to-high levels of implementation for the whole-school intervention. The 
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remaining four schools all had moderate levels of implementation, taking into account all 


components of the whole-school professional development. 


It should be noted that the moderate-to-high average levels of participation are related more to the 
high levels of whole-school coaching than to high levels of teacher participation in the group 
training. Although attendance at large-group professional development training was poor, multiple 
in-school coaching visits were made to each individual school, offering teachers opportunities to 


learn the techniques taught at the large-group training in an individualized way. 


C.3 Summary of the Whole-School Intervention Impact Findings 


All 19 Striving Readers schools were eligible to receive the whole-school intervention. As a 
compatison group, school-level data were collected for all other middle schools in the district. The 
unit of analysis for this study was the school. Students in grades 6-8 who took the NJASK in the 


district were eligible and were included in school-level data for the analyses. 


After 3 years of implementation, results from the analysis of whole-school NJASK outcomes 
indicate no difference in performance levels between the whole-school treatment group and the 
compatison group on the Language Arts section of the state assessment. Low teacher participation 
in the large-group professional development training as well as high teacher turnover could 


potentially explain the lack of program impact. 
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Introduction and Study Background 1 


L.A District Overview 


One of the oldest school systems in New Jersey, the Newark Public School (NPS) system dates back 
to 1676. Barringer High School in Newark’s North Ward is the third oldest public high school in the 
nation. Currently serving a student population of 39,440, NPS is also the largest school district in 
New Jersey. That student population is diverse: approximately 58 percent African American, 33.5 
percent Hispanic, 7.5 percent Caucasian, and 1 percent Asian or other heritage. Approximately 10 
percent of the students are English language learners (ELL), and 14 percent receive special 
education services. Analysis of district achievement data reveals that students in the middle grades 
are struggling in the area of language arts. In the spring of 2009 only 39 percent of 6th graders, 37 
percent of 7th graders, and 52 percent of 8th graders were proficient on the state reading 


assessment. 


The existing literacy curriculum for the middle grades uses the New Jersey Core Content Curriculum 
Standards for literacy instruction and incorporates research-based strategies from the National 
Reading Panel (2000) to bolster the acquisition of phonemic awareness, phonics, fluency, 
vocabulary, comprehension, and motivation. In the district’s existing curriculum, students have 
extended learning time; have the opportunity to read high interest, age-appropriate materials; and 
work in small groups to maximize cooperative learning. Daily reading instruction must be at least 90 
minutes. For grades 6, 7, and 8, the literacy curriculum primarily emphasizes comprehension and 
vocabulary and uses the textbook, The Language of Literature McDougal Littell, 2002). The curriculum 


also uses a number of supplementary materials, including the following: 


7 Bridges to Literature—designed to maintain and build students’ comprehension 
through a reciprocal-teaching approach. The materials include a textbook, an 
assessment book, and a student workbook. The textbook contains a leveled reading 
series designed to meet the needs of delayed readers by providing them with accessible, 
high-interest, on-level instruction. The assessment book and Bridges Skill Builder 
workbook provide assessments, writing prompts, and revising and editing activities to 
enhance students’ literacy skills. 


. Classtoom-Leveled Libtraties—provide students with continued opportunities to read 
high-interest and age-appropriate materials that build vocabulary, fluency, and 
comprehension. 


The literacy curriculum is designed to ensure that students learn the essential skills of listening, 
reading, speaking, reading, writing, and spelling. Furthermore, the goal of the curriculum is to enable 
students to become contributing members of society by helping them develop and expand their 
view of themselves and the world, as well as obtain necessary skills such as critical thinking, problem 


solving, and creativity. 


Adolescent Literacy in Context 


Numerous researchers have categorized adolescent literacy in the United States as being in a state of 
crisis (Fisher & Ivey, 2006; Jacobs, 2008; Kamil, 2003). Adolescent reading and writing skills are on 
the decline (Carnegie Council, 2010) and recent data from the National Assessment of Educational 
Progress (NAEP) show that 34 percent of 4th grade and 26 percent of 8 grade students in the 
United States are reading below the basic level (U.S. Department of Education, 2010). Among the 
students in large cities, 46 percent of 4" graders and 37 percent of 8" graders ate reading below the 
basic level (U.S. Department of Education, 2010). These data indicate that large numbers of middle 
school students are at a significant disadvantage before reaching high school, placing these students 
at tisk for poor academic outcomes. Moreover, students with poor literacy skills are more likely to 
drop out of high school or graduate without the basic skills needed to attend postsecondary 


education, succeed in the workforce, or act as an informed citizen (Carnegie Council, 2010). 


This concern over the low literacy achievement of adolescents has focused the attention of 
educators, policymakers, and researchers on finding effective ways to intervene with struggling 
readers. Literacy intervention is often targeted towards students in the early stages of schooling; 
however, when students reach middle school, reading gets more complex as they are expected to 
read expository text across a variety of content areas, decipher complex passages, synthesize 
information, and form independent conclusions based on data (Carnegie Council, 2010). Recent 
testing shows that early performance and gains in literacy often dissipate as students move through 
middle school, indicating that early literacy intervention does not inoculate students against later 
literacy failure (Carnegie Council, 2010). These findings highlight a need to identify and implement 
effective, research-based interventions for improving the literacy skills of middle school students. 
Currently, the research base evaluating adolescent literary interventions is limited, and there is little 
understanding of which interventions are likely to be effective (Slavin, Cheung, Groff, & Lake, 
2008). 


In response to the poor literacy achievement of middle school students as well as the dearth of high- 
quality research examining the effectiveness of middle school literacy programs, the U.S. 
Department of Education awarded eight Striving Readers grants in 2005. The goal of these grants is 
to improve the literacy skills and achievement of struggling readers in middle and high school and to 
build a scientific research base for interventions that improve adolescent literacy skills. The grant has 
two parts: a targeted intervention and a whole school intervention. The targeted intervention uses a 
randomized control trial design to test the effectiveness of an intervention (READ 180) on the 
subset of student with the poorest literacy skills, while the whole school intervention is designed to 
improve the literacy skills of all students by supporting the effective implementation of the existing 
district curriculum. NPS was among the eight school districts to receive the grant. Scholastic’s 
READ 180 Enterprise Edition* was chosen to replace the district’s existing language arts curriculum 
for the targeted portion of the grant. READ 180 is designed to enhance reading and comprehension 
skills of struggling readers in the upper elementary, middle, and high school years. READ 180 was 
one of four middle and high school literacy curricula identified by Slavin and colleagues (2008) as 
having “moderate evidence” of effectiveness. Furthermore, the curriculum includes several of the 
strategies identified as effective including explicit vocabulary instruction, comprehension strategy 
instruction, the use of high interest or relevant reading materials, and the use of a mixed methods 
instructional mode (Slavin, et al. 2008; What Works Clearing House, 2008). The whole school 
intervention includes two professional development models delivered by New Jersey City University 
(NJCU) and National Urban Alliance (NUA). These developers offer large-group training for 
teachers on ways to effectively promote literacy in the classroom. The two developers also offer in- 


school coaching support to teachers. 


1.B Description of the Intervention Models 
1.B.1 Targeted Intervention 


Scholastic’s READ 180 Enterprise Edition was chosen to replace the district’s existing language arts 
curriculum for targeted intervention.” READ 180 directly addresses the individual needs of 
adolescents reading below grade level by using adaptive and instructional software, high-interest 


literature, and direct instruction. READ 180 also includes an assessment component: the Scholastic 


? READ 180 for NPS is a 3-year intervention. The instruction has remained the same for all 4 years of the study. 


3 A description of the existing district curriculum is provided in section 1.A. 


Achievement Manager (SAM), which collects and organizes student performance data for 


supporting data-driven instruction. 


The READ 180 instructional model provides a straightforward way to organize instruction and 
classroom activity. The instructional model consists of a 90-minute literacy block that begins and 
ends with whole-group, teacher-directed instruction (20 and 10 minutes, respectively). During the 60 
minutes between the whole-group sessions, students break into three small groups that rotate among 


three stations (Figure 1). 


Figure 1. READ 180 instructional model 
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During the first 20-minute session, the READ 180 teacher instructs the entire class of no more than 
21 students. During the next 60 minutes, the students break into smaller groups of equal size, which 


proceed through the following three 20-minute rotations: 


1. Small-group instruction—The teacher sits with this small group to provide direct and 
explicit instruction on reading comprehension strategies using the tBook.* 


2. — Independent reading—Students enter a comfortable seating area where they read leveled 
paperbacks with the option of adding audio through headphones as modeled reading. 


4 In the commercial Scholastic model, the small group portion of the lesson is devoted to direct instruction using READ 180’s rBooks only. 
However, in a modification to the standard curriculum, NPS has supplemented these rBooks with books from McDougal Littell. These materials 
are used for independent reading of leveled texts and for teaching the use of graphic organizers and specific vocabulary. The district opted to 
incorporate the McDougal Littell series in READ 180 classrooms as an additional resource for exposure to literature. 


3. — Software use—Nine topical CD-ROMs provide students with background knowledge and 
mental models through full-motion video. Students encounter a reading passage based 
on the video that is at the appropriate ability level of that student. After the video and 
passage, students proceed through three zones: 


Word zone—Instruction for developing basic decoding skills 
Spelling zone—Instruction in the acquisition and transfer of spelling patterns and sounds 


Success zone—Assessment for comprehension, word recognition, and fluency 


In the last 10 minutes of class, the teacher provides a whole-group wrap-up. 


READ 180 was originally designed for one complete year of instruction. According to Scholastic, 
with the purchase of add-on materials, instruction can last up to 3 years. NPS is implementing 
READ 180 for 3 years and developed a pacing guide (Appendix A) in Year 2 that outlines each 3- 
week plan of instruction, with four stop points built in to analyze report data to determine 
differentiated instruction needs. The first 2 days of this time are spent on prereading activities, such 
as building a background in the subject area with anchor videos and previewing vocabulary. The 
next 6 days are spent on reading strategies, including teaching, practicing, and applying the main idea 
and details. Days 9 and 10 are spent on reviewing and extending vocabulary, with Days 11-13 


focusing on writing and grammar. Functional literacy is covered the last day before wrap-up. 


1.B.1.1 Participating Schools, Teachers, and Students 


For the targeted intervention, eligible middle level schools in NPS were identified based on the 


following criteria: 


: Be Title I eligible 
: Serve a minimum of two grades (6, 7, 8) 
7 Not already using READ 180 


. Be categorized as “school in need of improvement” (INOJ) under the No Child Left 
Behind Act 


a Serve a minimum of 25 eligible students 


Based on these criteria, 20 schools were eligible to participate in the targeted intervention. These 


schools then were randomly assigned to treatment and control groups, with the treatment schools 


slated to receive the READ 180 curriculum. After the random assignment had taken place, two 
schools in the control group merged, leaving 10 schools in the treatment group and 9 schools in the 
control group. The 19 participating schools serve predominately minority populations and almost 
half of the students (49 percent) are eligible to receive free and reduced meals. Table 1 shows 
demographic data from all grades at participating schools at the beginning of the first year of the 


Striving Readers intervention. 


Table 1. Demographic characteristics of participating Striving Readers schools (2006-2007) 


% Free & 
Grades No. of No. of % African % % reduced 

School served students teachers %Asian American’ Hispanic Caucasian lunch 
School 1 K-8 506 44 0.2 96.0 3.8 0 70.0 
School 2 K-8 319 30 0 97.5 2.2 0.3 32.0 
School 3 PK-8 292 35 0 73.6 26.0 0.3 75.0 
School 4 3-8 372 32 0 97.6 2.4 fe) 31.4 
School 5 PK-8 446 45 0 78.5 21.5 (0) 68.6 
School 6 K-8 602 48 ¢) 47.7 52.2 0.2 44.0 
School 7 K-8 790 66 1.6 30.8 65.7 1.9 39.7 
School 8 PK-8 337 34 0 92.3 7.7 0 77.4 
School 9 PK-8 594 49 0 45.1 43.9 10.9 49.2 
School 10 K-8 349 341 0 95.1 4.9 0 37.8 
Schoolit 5-8 753 50 0.9 21.0 77.0 0.8 56.7 
School 12 K-8 572 56 0.2 97.7 2.1 0 45.5 
School 13 PK-8 754 66 0.7 42.6 55.2 1.6 43.6 
School 14 PK-8 515 45 0.2 46.4 53.2 0.2 48.0 
School 15 PK-8 1041 71 8.7 70.2 13.6 6.6 28.5 
School 16 PK-8 464 47 fe) 93.1 6.7 0.2 61.2 
School 17 PK-8 776 56 0.1 29.6 68.4 1.8 38.8 
School 18 K-8 776 56 3.5 11.0 83.6 1.9 31.4 
School 19 PK-8 679 68 (0) 94.4 5.7 0.1 56.3 
Average 576 49 0.8 66.3 31.4 1.4 49.2 


READ 180 teachers in the treatment schools were selected by their school principals. When filling 
classroom positions in their schools, principals typically have control over whom they select for 
certain assignments. Thus, the principals selecting READ 180 teachers were following their typical 


placement procedures. 


For students to be eligible for the targeted intervention in Year 1, they had to be enrolled in one of 
the eligible middle schools and be in grades 6, 7, or 8. Furthermore, students’ eligibility was based on 
their score on the reading subtest of the New Jersey Assessment of Skills and Knowledge (NJASK). 


In New Jersey, anyone scoring below a 200 is considered “partially proficient,” which is the lowest 
category possible. Scores from 200 to 249 are “proficient,” and scores above 249 are “advanced 
proficient.” The following cut-off scores for student eligibility were set by the district, based on one 


standard deviation” from the norm: 


6th grade = 198 
7th grade = 186 


8th grade = 192 


In Years 2, 3, and 4, additional 6th graders were added to the sample. As in Year 1, the cut-off score 
of 198 was applied to the new cohorts of 6th graders. In Year 5, no new cohort of students was 
added and only the 8" grade students were followed. Transfer students without an NJASK score 
were not eligible to participate in the evaluation. Table 2 shows the number of students by program 


year and selected demographic characteristics. 


1.B.1.2 Professional Development Model and Ongoing Support 


Professional development and support of the targeted intervention are critical to implementation. 
Professional development has been offered to classroom teachers, literacy coaches, principals, 
technology coordinators, and Resource Teacher Coordinators (RTCs) to ensure all have a clear 
understanding of what is being taught and the strengths and challenges in the READ 180 program 
implementation. Professional development is provided by Scholastic and is supported through in- 


school coaching visits. 


Teachers 
In Year 4, new teachers were invited to attend two large-group training days (5.5 hours each). Five 


teachers attended at least 1 day of training. Returning READ 180 teachers did not attend the training 


because the training is geared towards new teachers and they had attended in previous years. 


5 As determined by NPS. 


Table 2. Characteristics of students in the evaluation of the targeted intervention 
Students in Students in All 
Number treatment schools control schools students 
(column %) Year1t Year2 Year3 Year4 /Year5 | Yeart Year2 Year3 Year4 /Year5 | Year1t Year2 Year3 Year4 Year5 
Total number of 708 648 638 577 170 663 584 529 493 144 1,371 1,232 1,167 1,070 314 
students (52%) (53%) (55%) (54%) (54%) (48%) (47%) (45%) (46%) (46%) 
Average no. of 78.7 64.8 63.8 57.7 17.0 66.3 64.9 58.8 54.8 16.0 72.2 64.8 61.4 56.3 16.5 
students per 
school 
Grade 
6th grade 37% 35% 35% 35% - 36% 31% 35% 38% - 37% 33% 35% 36% - 
7th grade 30% 36%* 32%* 34%* - 35% 37% 30% 32% - 33% 36% 31% 33% - 
8th grade 33% 29%* 32%+ 31%+ 54%+ 28% 32% 35% 30% 46% 30% 31% 34% 30% 100% 
Gender 
Male 57% 56% 59% 60% 66% 55% 51% 55% 63% 65% 56% 54% 57% 61% 65% 
Female 43% 44% 41% 40% 34% 45% 49% 45% 37% 35% 44% 46% 43% 39% 35% 
Status 
Economically 91% 59% 62% 62% 91%: 84% 57% 58% 61% 83% 88% 58% 60% 62% 88% 
disadvantaged 
ELLs 6% 8% 11% 15% 12% 8% 7% 9% 11% 11% 7% 8% 10% 13% 12% 
Special education| 39% 44% 48% 46% 42% 40% 40% 39% 42% 37% 39% 42% 44% 44% 39% 
Race/ethnicity 
African American 57% 57% 54% 53% 49% 58% 54% 54% 57% 53% 58% 55% 54% 55% 51% 
Hispanic 41% 41% 43% 43% 46% 41% 44% 43% 41% 45% 41% 43% 43% 42% 45% 
Caucasian <1% 1% 1% 1% <1% 1% 1% 1% 1% - 1% 1% 1% 1% <1% 
Other <1% 1% 3% 2% 1% <1% 1% 1% 1% 2% <1% 1% 2% 1% 2% 


Not identified 


* Two years of exposure. 


+Three years of exposure. 


= The economic status was not provided for 322 students (178 treatment students and 144 control students). 


* The economic status was not provided for one student. 


The amount of training declined from Year 1 (3 days; 16.5 hours) to Year 2 (2 days; 8 hours) and 
again in Year 3 (1 day; 5.5 hours). Table 3 provides more detailed information about hours of 
training by year and reasons for changes to the training model. In Year 3, Scholastic offered 1 day 
(5.5 hours) of large-group training. Only one READ 180 teacher (a teacher new to Striving Readers) 
attended this training. The other teachers new to READ 180 were not yet hired at the time of the 


training and, therefore, could not attend. 


Consultants gave teachers appropriate background information on READ 180 and the research 
supporting its development. Teachers had opportunities to gain hands-on experience using the 
READ 180 curriculum and were trained to use Scholastic’s tools to aid the implementation of the 
curriculum and management of their classroom. In addition, READ 180 consultants trained teachers 


on how to use assessment results to inform instruction. 


READ 180 teachers receive ongoing classroom support in addition to the large-group training. 
Representatives from Scholastic’s READ 180 program conducted these support visits on an as- 
needed basis in Years 1 and 2. Each of the nine treatment schools was visited at least once by a 
consultant from Scholastic in Years 1 and 2. In Years 3 and 4, teachers received classroom support 
from district RTCs. These visits consisted of providing technical assistance to teachers, monitoring 


the program, and ensuring that the model is being implemented correctly. 


Overall, changes in the number of hours dedicated to training decreased from Years 1 to 3 of the 
grant but increased in Year 4. See Table 3 for changes in professional development over the four 


years and reasons for changes 


Literacy Coaches 


Literacy coaches are housed in each school in the district. In general, their role is to support 
classrooms teachers in all grades as they implement the district curriculum. Because literacy coaches 
are not supported by the Striving Readers grant, they did not have a specific role in the program. 
However, literacy coaches in the treatment schools have been strongly encouraged to attend the 
training provided for the READ 180 teachers. 


Table 3. 


Changes to READ 180 professional development model 


READ 180 PD Model 
Year1 Year 2 Year 3 Year 4 Reason for change 

Teacher 

training 

Large-group 

Newteachers 3days(16.5 2 days (8hrs) 1day(5.5 2 days (11 To build internal capacity, 

hrs) hrs) hrs) district RTCs played a 

larger role in supporting 
READ 180 teachers from 
Years 1 to Year 3 with in 
classroom support and 
less large-group training. 
Training increased in Year 
4 due to staffing changes. 

Returning NA 2 days (8hrs) Ohrs 0 hrs To build internal capacity, 

teachers district RTCs played a 
larger role in supporting 
READ 180 teachers from 
Years 1 to Year 4 with in 
classroom support and 
less large-group training. 

Classroom 

support 

All teachers District RTCs: District RTCs: District RTCs: District RTCs: No change 


Administrator 
support 


Training for 
principals on 
READ 180 


Technology 
support 


Training for 
technology 
coordinators 


as needed 
basis 


Observations 
by principals 


2 hrs 


1% day 


as needed 
basis 


Observations 
by principals 


2 hrs 


1% day 


as needed 
basis 


Observations 
by principals 


O hrs 


0 hrs 


as needed 
basis 


Observations 
by principals 


0 hrs 


0 hrs 


Communication between 
RTCs and administrators 
became stronger; RTCs 
met with administrators 
periodically to analyze 
SAM reports and fidelity 
to the model. 


NPS continued to build 
internal capacity by 
having the systems 
analyst provide support in 
Year 4. 
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Principals 


Each year, principals of the treatment school are expected to periodically review SAM reports and 
analyze the strengths and needs of students. They are furthermore expected to observe READ 180 
classrooms to help ensure fidelity. Administrators are also asked to communicate with the Office of 


Language Arts Literacy any concerns they have with READ 180 or classroom instruction. 


Principals recetved 2 hours of training from Scholastic on the READ 180 model in Years 1 and 2. 
This training included the structure and management of a READ 180 classroom, use of curriculum 
materials, and how to differentiate instruction based on data from Scholastic Reading Inventory 
(SRI) reports. Principals did not receive this training in Year 3 or Year 4. However, principals are 


updated regularly by the RTCs on READ 180 implementation. 


Technology Coordinators 


Each year, technology coordinators actively monitor the READ 180 equipment and troubleshoot 
technical issues as needed. They also are responsible for creating student passwords, inputting 
student information at the beginning of the year, activating student site licenses, and creating class 
rosters on SAM. In Years 1 and 2, all nine treatment school technology coordinators received one- 
half day of training from Scholastic so that they could better support the installation and operation 
of the technology component of the curriculum. This training was not provided in Years 3 and 4. 
However, the district’s systems analyst communicated regularly with the technology coordinators to 


provide ongoing technical support, as needed. 


RTCs 


Five RTC positions are supported by the Striving Readers grant. In Year 1 all five positions were 
filled, and the RTCs were tasked with providing support to teachers for both the whole-school 
intervention and the targeted intervention. The number of RTCs decreased to three active RTCs in 
Year 4 due to extended maternity leave and a medical leave of absence for two RTCs. RTCs have 
attended teacher READ 180 trainings and have visited all READ 180 classrooms to provide support 
to teachers via activities such as conducting needs assessments; providing demonstration lessons, in- 


class support, and coaching; assisting with instructional plans; conducting READ 180 articulation 
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meetings; maintaining accurate records; interpreting student assessment data; and serving as liaisons 


with the district administration. 


1.B.2 Whole-School Intervention 


The whole-school intervention is designed to support the effective implementation of the existing 
district curriculum. Its goal is to improve students’ ability to “read to learn” across multiple content 
areas. To this end, the intervention provides professional development to improve the literacy 
instruction of language arts and content area teachers. This professional development is provided 
through large-group training and is supported by in-school coaching visits. Language arts teachers 
and literacy coaches receive training from New Jersey City University (NJCU), and content area 


teachers receive instruction from the National Urban Alliance (NUA). 


Language Arts Teachers 


The professional development course for language arts teachers was designed by the literacy faculty 
from the NJCU School of Education. After receiving training, teachers are expected to implement 


the following strategies in their classrooms: 


. Use graphic organizers, including flowcharts, webs, and tables (e.g., K-W-H-L-S) to 
build student reading comprehension skills 


= Establish routines for effective oral and silent reading 
= Use model text annotation, note taking, and post-reading reflection 


= Use anticipation guides, the SQ3R (Survey, Question, Read, Recite, Review) method, 
and double-entry journals to build student writing, fluency, and reading comprehension 


skills 
. Use small groups to target and differentiate instruction 
: Use model context clues and personal dictionaries to enrich vocabulary and build 


linguistic competence 


| Guide student discussion and use brainstorming techniques to facilitate students’ 
exploration of the connections between reading and writing 
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= Review student work samples, including portfolios, journals, and notebooks to show the 
use of graphic organizers 


NJCU provides in-school coaching visits during the school year. The visits are tailored to assist 
teachers through modeling and discussing classroom strategies learned at training. In alignment with 
the work of NJCU, the Striving Reader RTCs provided additional support through in-school visits, 
beginning in September and ending in June. During these visits, RTCs offer classroom suppott, 
coaching, and modeling; assistance with student work; and assistance using student data to inform 
instruction. In addition, the RTCs assist teachers in preparing for the NJASK, the General 


Education Provisions Act (GEPA) initiative, and standards-based lesson planning. 


Content Area Teachers 


Content area teachers in Striving Readers schools receive training and in-school support from NUA, 
a nonprofit professional development group known for its work in content literacy. The 
instructional strategies undertaken by language arts literacy teachers are expected to improve 
students’ vocabulary, fluency, and reading comprehension skills. To build on these improved skills, 
math, social studies, and science teachers are expected to incorporate NUA-developed graphic 


organizers (“Thinking Maps”), including the following: 


. Circle Maps for context description 
. Double Bubble Maps to compare and contrast information 
: Tree Maps for inductive and deductive classification 


. Brace Maps to identify part-whole relationships 
. Flowcharts to review sequential order 
7 Multiflow Maps to explicate cause and effect relationships 


2 Bridge Maps to interpret analogies and metaphorical concepts 


Based on NUA professional development, math, social studies, and science teachers also use 
anticipation guides to model brainstorming and prewriting strategies, as well as use taxonomies to 
promote word study and vocabulary development. Additionally, NUA mentors provide in-school 
coaching visits during the school year. These visits are tailored to demonstrate (and provide 


coaching in) the application of the strategies presented during the large-group workshops. As with 
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language arts teachers, the RTCs provided additional support to content area teachers in the form of 


in-school visits. 


1.B.2.1 Professional Development 
Language Arts Teachers 


NJCU’s professional development was designed to support the district’s core literacy program, with 
a focus on strategies that enhance vocabulary development, fluency, and reading comprehension 
such as graphic organizers and flowcharts. The amount of large-group training provided for 
Language Arts teachers decreased from previous years. In Years 1, 2, and 3 of the project, NJCU 
provided a summer institute. This institute was cancelled in Year 4. In Years 2 and 3, NJCU also 
provided two large-group training days during the school year. In Year 4, only one large-group 
training was offered. The decrease in training was due to teaching days being added to the school 
district calendar (taking away from the number of days available for professional development) and 


budget constraints. 


In addition to the large-group training described above, NJCU teachers received ongoing classroom 
support in the form of in-school coaching visits. NJCU coaches were expected to visit all 19 Striving 
Readers schools, starting in September 2009 and ending in May 2010. Each school was to be visited 
by an NJCU coach 10 times® in Year 4. The number of coaching visits also decreased from previous 
years, where schools were to receive 12 total visits. Again, the reason given for the decrease was 
budget constraints. See Table 4 for a description of changes to the professional development 


trainings over the four years of the grant. 


Content Area Teachers 


The number of large-group training sessions for content area teachers also decreased in Year 4. In 
Years 1-3, math, science, and social studies teachers were offered a summer institute and two large- 
group workshops during the school year. In Year 4, one day of large-group training was provided. 


The reasons for the decrease are highlighted in Table 4. Overall, training decreased because teaching 


6 As contracted through the Striving Readers grant, NJCU was expected to make five visits to Striving Readers schools in Year 4; however, Title I 
funds were used to subsidize the number of visits for a total 10 visits per school. 


14 


Table 4. 


Changes to professional development models 


Large-group 
Teacher 
Training 
Summer 
Institute 


School Year 


Classroom 
support 


Large-group 
Teacher 
Training 
Summer 
Institute 


School Year 


Classroom 
support 


4 half 
days 
(16 hrs) 
O days* 


NJCU 
coach: 
Each 
school to 
receive 5 
visits. 


3 half 
days 
(12 hrs) 
2 days 
(11 hrs) 


NUA 
Mentor: 
Each 
school to 
receive 
15 visits. 


4 half 
days 
(16 hrs) 


3 days 
(17.5 hrs) 


NJCU 
coach: 
Each 
school to 
receive 
10 visits. 


3 half 
days 
(12 hrs) 
2 days 
(12 hrs) 


NUA 
Mentor: 
Each 
school to 
receive 
15 visits. 


4 half 
days (16 
hrs) 


2 days 
(12 hrs) 


NJCU 
coach: 
Each 
school to 
receive 
12 visits. 


3 half 
days 
(12 hrs) 
2 days 
(12 hrs) 


NUA 
Mentor: 
Each 
school to 
receive 
15 visits. 


O Days 


1 day 
(6 hours) 


NJCU 
coach: 
Each 
school to 
receive 12 
visits. 


O Days 


1 day 
(6 hours) 


NUA 
Mentor: 
Each 
school to 
receive 15 
visits. 


Due to budget cuts, a summer institute 
could not be offered in Year 4. 


Days during the school year were added 
in Year 2 to establish parity with NUA. 
Reduction from Year 2 to Year 3 and 
from Year 3 to Year 4 was because of 
reduction in district-allocated 
professional development days. 


In Years 2 and 3 more visits were added 
to establish parity with NUA. 


Due to budget cuts, a summer institute 
could not be offered in Year 4. 


Reduction in Year 4 was because of 
reduction in district-allocated 
professional development days. 

No Change 


* Two make-up days were provided in September of Year 1 for teachers who missed the summer training. 


Table 4. Changes to professional development models (continued) 


Large-group | 4 days O days O days O days RTCs provided professional 

Teacher (22 hrs) development in Year 1 because NJCU 
Training did not provide large-group training 
School Year during the school year. As NJCU’s 


involvement intensified (large-group 
training and increased number of site 
visits), it was no longer necessary to 
have RTCs present. 


Classroom District District District District No Change 
support RTC: As RTC: As RTC: As RTC: As 

needed needed needed needed 

basis basis basis basis 


days were added to the District calendar (taking away from the number of days available for 
professional development) and budget constraints. Despite the decrease in training days, the intent 
of the training remained the same: to train teachers in cognitive strategies that focus on the teaching, 
learning, and assessment of advanced thinking; to break down school isolation; to build effective 


school teams; and to create a community of learners. 


In addition to the large-group training described, NUA teachers received ongoing classroom support 
through in-school coaching visits. The plan was for NUA to visit all 19 Striving Readers schools in 
the 4th year of the grant, starting in September 2009 and ending in June 2010. Each school was to 
be visited by a NUA mentor for 10 days. The number of in-school coaching visits decreased from 


previous years again due to budgetary reasons (See Table 4). 


1.B.2.2 Participating Schools, Teachers, and Students 


The 19 schools participating in the targeted intervention (see Section 1.B.1.1 for eligibility criteria) 
are also the schools participating in the whole-school intervention. However, the whole-school 
intervention is not being evaluated with a randomized design, so there ate no treatment and control 
schools. For the whole-school intervention, all 6th, 7th, and 8th grade teachers and literacy coaches 


in all 19 schools receive the intervention. 


In Year 4, 337 teachers were eligible to recetve professional development as part of the whole- 


school intervention. Of these, 138 were eligible for professional development provided by the NUA 
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(teachers who taught on/y math, science, or social studies). Fifty-nine’ other teachers were eligible for 
training from NJCU (teachers who taught ov/y language arts). In addition, 140 teachers were eligible 
for both NUA and NJCU training. These teachers either taught both language arts and a content 
area subject (usually social studies) or they taught all subjects (usually special education or bilingual 
teachers). Table 5 provides the number of teachers eligible for the professional development 


sessions for the two whole-school interventions, by subject areas taught. 


Table 5. Distribution of teachers by subjects taught in Year 4 


NUA NJCU 
No. of No. of No. of No. of 
Teacher subject teachers schools teachers schools 
Content area only (NUA) 138 19 N/A N/A 
Language arts only (NJCU) N/A N/A 59 19 
Content area & language arts 140* 19 140* 19 
Total 278 19 199 19 


* These teachers are counted in both categories. 


Students in the whole-school intervention are from all 19 Striving Readers schools. All 6th, 7th, and 


8th grade students from these participating Striving Readers schools are included. 


1.C Logic Models 
1.C.1 Targeted Intervention 


The targeted intervention provides language arts for middle school students through direct 
instruction, instructional software, and literature. It also replaces the regular language arts 
curriculum. The theory of change that underpins READ 180 is displayed in Figure 2. This theory of 


change provides the conceptual framework for the evaluation. 


The first two columns in Figure 2 describe the resources necessaty to implement the intervention. 
The first column lists the materials and resources that should be in place to support full 
implementation and use of READ 180. Materials include leveled library books, student rBooks, and 
Flex books, while the SAM database system allows teachers to periodically review and analyze the 


strengths and needs of students. The second column includes the professional development and 


7 Includes 19 literacy coaches. 
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Figure 2. Targeted intervention logic model® 


Inputs 


Materials /Resources Professional 


Development/Support 
Computers and adaptive & : oe 


instructional software Teachers & Literacy Coaches 


CDs for independent reading © =63 half days of whole-group training, or 
half day of make-up training. 


High-interest literature -READ Provided by Scholastic. 


180 paperback library in each 
classroom day of whole-group training on using 
student data to drive differentiated 
instruction. Provided by Scholastic. 


READ 180 rBooks 
(supplemented by District 
curricular materials day of whole-group training on 
interpreting READ 180 data reports. 
READ 180 Flex books Provided by Scholastic. 

Scholastic Achievement Teachers Onl 
Manager (SAM) — management 

system for READ 180 software ° n-classroom support from district 
rograms RTCs and school literacy coaches on 
an as-needed basis* 


Scholastic technical assistance 
(as needed) n-classroom technical assistance from 


nt Scholastic on an as-needed basis 
District director of language arts 


& literature Principals 
District project manager ° half day of traming from Scholastic 


District resource teacher 
coordinators (RTCs) 


READ 180 systems analyst 


Technology Coordinators 
° half day of traming from Scholastic 


n-school literacy coaches 


n-school technology 
coordinators 


Classroom observers (Westat) 


Contextual effects such as the characteristics of the school distri 


Activities 


Daily 90-minute instructional block. 


20-minutes whole-group instruction 
to start the class. 


Small-group rotations in which 
students are divided into groups and 
spend 20 minutes in each zone: (1) 
small-group instruction, (2) modeled 
and independent reading, and (3) 
READ 180 topic software 


10 minutes of whole-group wrap-up 
to conclude the class. 


Teachers regularly use diagnostic 
tests (SRI) and Scholastic 
Achievement Management for 
continuous assessment, placement, 
and monitoring 


No more than 21 students per class 


Regular use of instructional strategies 
and materials contained in READ 
180 program guides supplemented 
with district text, including 
independent reading of leveled texts, 
use of graphic organizers, and 
teaching of specific vocabulary. 


Student enrollment for the entire 
school year. 


Instruction follows rBook scope & 
sequence 


Short-Term Long-Term 


Outcomes Outcomes 


Improved student reading Improved student 
skills reading skills 


Improved attendance 


Improved student 
engagement and behavior®® § ----------£--------- 7 


® = Improved 


1 1 

: wt 

@ Decrease in number of 3 ; 
roy achievement across all 


1 

1 

1 

disciplinary incidents** : H 
sciplinary incidents L subject areas ! 

® = Improved literacy ! i 
1 


! 
1 
‘ . l 
instruction** | Bee eee eee eee 
1 
1 


ther instructional programs in use, and external events may also influence outcomes. 


*The RTC school visits supported both the whole-school and targeted mterventions of the Newark Striving Readers program. 


These outcomes are not directly measured under the Newark, Striving Readers grant 


8 Logic model from Year 1 of the study. 


support services’ necessaty for implementation. Scholastic staff, RTCs, literacy coaches, and 


technology coordinators all support the intervention. 


The third column describes the activities of the intervention and includes instructional strategies that 
are necessary for full implementation of the targeted curriculum. All READ 180 teachers are 
intended to receive large-group training from Scholastic. Additionally, make-up training or in-school 


support is intended for those teachers who missed the summer session. 


The last two columns of the logic model provide the short- and long-term outcomes that are 
anticipated. The theory of change posits that when all of the necessary resources are in place and the 
appropriate teaching and learning activities occur, students will first demonstrate improved reading 
skills and improved classroom behavior. The theory of change then suggests that these short-term 
outcomes will, in turn, result in longer term effects, reflected in improved achievement test results, 
increased school attendance, decreased discipline problems, and gains in student learning in all 
subject areas (White & Haslam, 2005). 


The logic model reflects activities that occurred in Year 1 of the study. Specific changes made to the 


program model in Year 3 can be found in Table 3. 


1.C.2 Whole-School Intervention 


The theory of action driving the district’s whole-school intervention is illustrated in Figure 3. 
According to this logic model, language arts literacy teachers (including literacy coaches) receive 
professional development from NJCU. NUA provides the professional development for 
mathematics, science, and social studies content area teachers. Striving Reader RTCs support the 
implementation of both NJCU and NUA professional development approaches through site-based 


demonstration lessons and coaching. 
The first column in Figure 3 documents the basic resources needed to fully implement the 
intervention, including professional resource books; The Language of Literature print and web-based 


materials; and in-school support from RTCs, NUA, and NJCU consultants. The second column 


documents NUA and NJCU instructional strategies. These classroom practices incorporate what 


° Professional development activities varied in Years 2 and 3 of the grant. 
P: & 
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Figure 3. 


Program Inputs/ Activities 


Striving Readers Resource Teacher 
Coordinators (RTCs) 


Striving Readers project manager 


Professional resource books purchased with 
grant funds. 


NJCU consultants 
NUA consultants 


School-based literacy coaches, math coaches 
and lead science teachers 


NJ Core Curriculum Content Standards and 
Curricular Frameworks in Reading and Writing 
for Grades 6-8 


Use of The Language of Literature print and 
web-based materials and formative assessments 
for students by McDougal Little 


Professional development in large group setting 
provided by NJCU and NUA for Striving 
Readers grade 6-8 teachers 


In-school professional development/support 
provided by NJCU and NUA (modeling and 
discussion of effective classroom practices) 


On site coaching provided by Striving Readers 
RTCs; coaching model includes in class 
modeling, lesson planning, student work review, 
demonstration lessons, lesson study/ design, and 
teacher observation 


Professional development for Striving Readers 
RTCs & school administrators provided by 
NJ CU to monitor formative assessment data to 


track growth of students, observe, and 
evaluate teachers, ensure program 
implementation 


10 Logic model from Year 1 of the study. 


Teacher knowledge, attitudes, and beliefs 


Whole-school intervention logic model+° 


Classroom Practices: Intermediate 


e@ LAL teachers undertake the following activities in accordance with the 
district’s literacy curriculum for middle grades: 


¥ Extended learning time: 90-minute language arts literacy period 


¥ Model how students should work in small groups to maximize 
cooperative learning through discussion of texts (Bridges) 


Provide opportunities for students to read high-interest, age- 
appropriate material (Classroom Leveled Libraries) 


Provide opportunities for students to practice using reading and 
writing skills they are learning 


LAL teachers also use the following researched-based strategies 
introduced by NJCU to enhance student literacy: 


¥ Use graphic organizers, including flowcharts, webs, and tables (eg, 
K'WHLs) to build student reading comprehension skills 


Establish routines for effective oral and silent reading 
Model text annotation, note taking, and post-reading reflection 


Use anticipation guides, the SQ3R method, and double-entry 
journals to build student writing, fluency, and reading 
comprehension skills 


Use small groups to target and differentiate instruction 


¥ Model use of context dues and personal dictionaries to enrich 
vocabulary and build linguistic competence 


Guide student discussion and use brainstorming techniques to 
scaffold students’ exploration of the connections between reading 
and writing 


Review student work samples, including portfolios, journals, and 
notebooks to show use of graphic organizers 


Math, science, and social studies teachers use NUA-developed graphic 
organizers (“Thinking Maps”) to build student reading comprehension 
skills, vocabulary, and fluency, including the following: 


Circle maps for context description 

Double Bubble maps to compare and contrast information 
Tree maps for inductive and deductive classification 

Brace maps to identify part-whole relationships 

Flow charts to review sequential order 

Multiflow maps to explicate cause and effect relationships 
Bridge maps to interpret analogies and metaphorical concepts 


Math, science, and social studies teachers use anticipation guides to 
access student knowledge and model brain storming and pre-writing 
strategies 


School administrators, Striving Reader RTCs, literacy coaches, math 
coaches, and lead science teachers monitor formative assessment data 
to track growth of students, observe and evaluate teachers, ensure 
program implementation 
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lOliincesitey 


Short Term 


© Teachers actively acquire 
and apply research-based 


literacy strategies in 
teaching practices 


Students demonstrate 
improved fluency, 
vocabulary, and 


comprehension skills 


¢ Teachers show 
improved instructional 
behaviors and attitudes 
towards teaching 


Improved student 
achievement in reading 
on state and district 
assessments 


Improved achievement 
across all subject areas 


Fewer students need 
literacy based 
interventions in high 
school 


Sustained achievement 
through high school 


Increased number of 
students graduating high 
school via state 
summative assessment 


literacy experts and practitioners recommend to help middle school students master basic reading 
skills: direct, explicit instruction in comprehension; modeling of reading and thinking strategies for 
comprehension; cooperative learning and discussion of texts among students; self-selected reading at 
students’ ability levels to build motivation; ongoing progress monitoring; writing; age-appropriate 


and diverse reading materials; and interdisciplinary, classroom-based efforts to focus on literacy. 


The last two columns of the logic model provide the anticipated short- and long-term outcomes. As 
with the targeted intervention, the theory of change posits that when all required resources are in 
place and the described activities and classroom practices occur, students will demonstrate 
improvements in literacy, including fluency, vocabulary, and comprehensions. In addition, teachers 
are expected to demonstrate improvements in instructional behaviors and attitudes toward teaching. 
Furthermore, as a result of these short-term outcomes, the theory of change suggests that longer- 
term effects will occur. The expected longer-term outcomes include improvements in student 
achievement test results, a reduction in the number of students needing literacy-based interventions 
in high school, sustained student achievement through high school, and increases in the number of 
students who graduate from high school. In addition, teachers are expected to continue to 


implement research-based strategies as part of their instructional repertoire. 


1.D Brief Overview of Key Evaluation Design Features 

1.D.1 Targeted Intervention 

1.D.1.1 Key Research Questions 

The theoretical model for READ 180 presents a series of short- and long-term outcomes. Short- 
term outcomes include improved reading skills and student behavior, while longer term outcomes 
include continued improvement in reading skills, increased school attendance and grade promotion, 


and decreased disciplinary incidents. Some of these claims will be tested with the research questions 


presented in this section. 
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The three primary research questions that motivate the study design for the targeted intervention are 


the following: 


1. Does READ 180 significantly improve the reading skills of targeted students? 
2. Does READ 180 significantly improve school attendance of targeted students? 


3. Do different types of students benefit from READ 180 in different ways? 


The evaluatilon will determine whether READ 180 has a demonstrable impact" and if it works 
better for some students than for others. These questions will be addressed statistically by 


compating students in treatment schools to students in control schools. 


1.D.1.2 Unit of Random Assignment 


The schools eligible to participate in the Striving Readers program were randomly assigned to either 
the intervention or a control condition in May 2006 (see Section 1.B.1.1 for school eligibility 
requirements). The targeted evaluation is, therefore, a randomized cluster design; no classroom- or 
student-level random assignment was involved. Although randomly assigning students would be the 
most statistically efficient design, it was not feasible for this study. One of the main constraints was 
the cost of implementation, which is largely determined by the number of participating schools. 
Additionally, there are contamination and spillover effects associated with student-level 
randomization. For example, teachers are likely to be aware that a colleague is delivering a special 
intervention, and this awareness may influence their behavior. Additionally, intervention and 
nonintervention students would interact, possibly closing the gap between their differences. In either 


case, the impact estimates would be biased toward zero. 


Fairness is another factor that argued for implementing the intervention at the school level. 
Principals may resist cooperating if only some of their teachers are provided with special training 
and materials. Even if principals allowed differential treatment within a school, there may be 
pressure to allow some practices to spill over into control classrooms, thus biasing impact estimates. 
There also would likely be pressure to allow students who “deserve” intervention to transfer (cross 


over) to treatment classrooms, again biasing the impact estimates. 


1 By impact, the authors mean the difference between outcomes observed for students receiving the treatment and what would have been observed for these 
same students had they not participated in READ 180. 
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Accordingly, the authors opted for a design that would randomly assign schools to the intervention 

group or to the control group. As stated previously, this design eliminates many of the threats to the 
study’s feasibility and validity. Moreover, to increase the precision of the estimates, the authors used 
a tandomized block design. The school-level variables used for blocking" (in order of priority) were 


as follows: 


1. | Number of eligible students 
2. | Number of years school has been identified as “in need of improvement” 
3. Number of eligible students whose home language is not English 


4. Number of eligible students with an Individualized Education Program (IEP). 


Using the number of eligible students as the primary blocking variable, schools were divided into 
three groups: large schools (more than 100 eligible students), medium schools (51—100 eligible 
students), and small schools (25-50 eligible students). Within these three strata, schools then were 
sorted by number of years in need of improvement under No Child Left Behind and then by home 
language and special education status. The baseline sample of schools was 20: 10 treatment and 10 
control. The baseline sample became 19 when two of the control schools merged during the 


summer of 2006. 


1.D.1.3 Key Measures for Student Outcomes 


The key measures of student outcomes were the Reading and Language Arts subscales of the SAT 
10" and school attendance (see Table 6). The Reading and Language Arts portion of the SAT 10 
comprises three subtests: Vocabulary, Reading Comprehension, and Language Arts. The SAT 10 is 


administered by Westat staff and school attendance is compiled from district administrative data. 


!2 Blocking variables will be included in the statistical model to estimate effects. 


13 Abbreviated battery. 
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Table 6. Key measures of student outcomes 


Measures Student 
SAT 10 
Vocabulary v 
Reading Comprehension v 
Language Arts v 
School Records (Attendance) v 


The Vocabulary subtest assesses concepts such as synonyms, multiple-meaning words, and use of 
context clues to decipher the meaning of unknown words. The Reading Comprehension subtest 
assesses students’ reading achievement using text read for enjoyment (e.g., fiction, poetry), text read 
for informational or expository purposes (e.g., science, textbook material), and everyday functional 
text (e.g., directions, labels, forms). The items in this section consist of increasingly complex reading 
passages, along with multiple-choice questions associated with each passage. There are six to nine 


passages, depending on grade level. 


The Language Arts subtest is divided into three sections. The first focuses on language mechanics, 
including capitalization, punctuation, and usage. The second focuses on language expression, 
including writing strategies and sentence structure knowledge. The final section of the language arts 
subtest also focuses on language expression but on a higher level than the previous section. Students 
analyze written passages for the assessment of how well they recognize extraneous information and 


descriptive language and the combining of simple sentences. 
Because READ 180 also claims to improve struggling readers attitudes toward school (Scholastic 
Research and Evaluation, 2008), the last student outcome is school attendance. Attendance was 


measured as the number of unexcused absences during the school year. This information is provided 


fot each individual student from school administrative records. 


1.D.2 Whole-School Implementation 
1.D.2.1 Key Research Questions 
There ate two main goals for the whole-school evaluation. The primary goal is to determine the 


short-term effect of the professional development on teacher attitudes and instructional behavior. 


To collect these data, a series of teacher surveys was used. The secondary goal is to determine 
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whether these potential changes in teacher attitude and behavior affect student achievement, using 


data from the New Jersey state reading assessment. 


These two goals are reflected in the following three primary research questions for the whole-school 


evaluation: 
{. Does participation in an ongoing literacy professional development program 
significantly improve the attitudes and instructional practices of middle school teachers? 
2. Do different types of teachers benefit from ongoing professional development in 
different ways? 
3. Do changes in teacher instructional practices significantly improve the reading skills of 


middle school students? 


1.D.2.2 Unit of Random Assignment 


There was no random assignment for the whole-school intervention. All 6th, 7th, and 8th grade 
teachers from all 19 Striving Readers schools were eligible to receive the whole-school professional 


development and in-school coaching visits. 


1.D.2.3. Key Measures for Student and Teacher Outcomes 


The key measures of teacher outcomes come from a survey of teachers. This survey measured basic 
demographic information and previous training experience of participating teachers. It also captured 
the degree to which the teacher felt supported by his or her institution as well as his or her job 
satisfaction. Teachers also had an opportunity to self-evaluate their effectiveness at delivering 
literacy concepts and to provide information on their instructional practices. This information also 
tapped into their attitudes about what is important and appropriate in the classroom. Finally, 
teachers provided information on how they used data on their students’ assessments to tailor their 
classroom instruction. The pre-survey was administered before the teachers received any Striving 
Readers training. Multiple post-surveys then were administered during Years 1 and 2 to capture any 


change after receiving training. 


The key measure for students’ outcomes in the whole-school intervention is the state literacy 
assessment: the NJASK. Students in Grades 6 and 7 were assessed with the NJASK, while Grade 8 
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students were assessed with the NJASK8. More information on both of these assessments is 


provided in the following sections. Table 7 lists the key measures of student and teacher outcomes. 


Table 7. Key measures of teacher and student outcome variables 


Measures Teacher Student 


Teacher Survey 
Perception of Institutional Support 
Job Satisfaction 
Self Evaluation of Effective Teaching 
Classroom Instructional Practices 


Student Assessment 


LYNN NNN 


Literacy Skills Assessment 
NJASK (6th & 7th grade assessment) v 
NJASKS8 (8th grade assessment) v 


Grades 6 and 7: New Jersey Assessment of Skills and Knowledge 


The language arts literacy portion of the NJASK for Grades 6 and 7 provides a variety of texts, 
illustrations, and activities integrated in such a way that encourages students to think, communicate, 
and create original work. The variety and sequence of the assessment tasks aim to engage and sustain 
student interest and clearly measure what students know and can do. In each of the assessments, 
students write their own text and analyze text provided for them. Item types include performance- 
based writing tasks and multiple-choice and open-ended reading tasks. The NJASK focuses on the 


following content clusters: 


: Work with or interpreting text (reading)—These tasks involve identifying main ideas or 
themes, identifying supporting details, following directions, paraphrasing, organizing 
text, and understanding the purposes for reading. 


7 Analyzing or critiquing text (reading)—These tasks involve enhancing understanding 
through questioning, clarifying, and predicting; predicting meanings; drawing 
conclusions; and forming opinions about text and author techniques. Students are asked 
to explain or identify fundamentals and nuances of textual conventions and literary 
elements. 


. Generating text (writing)—These tasks involve the use of pictures or text to make 
decisions, solve a problem, or write a story, thereby generating original student work. 
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NJASK data are reported as scale scores ranging from 100 to 300 and are broken down into three 


proficiency levels: 


Advanced Proficient 250-300 
Proficient 200-249 


Partially Proficient 100-199 


The scores of students in the Partially Proficient category are considered to be below the state 


minimum for proficiency. 


Grade 8: New Jersey Assessment of Skills and Knowledge Grade 8 


The NJASK8 measures student ability in the areas of language arts literacy and is used to indicate 
progress students are making toward mastering skills they will need to pass the High School 
Proficiency Assessment. These skills are outlined in the state Language Arts Literacy Core 
Curriculum Content Standards and cover the content clusters in the NJASK. The assessment is 


designed to help students ask questions, speculate, explore new ideas, and form tentative opinions. 


The language arts literacy portion of the assessment focuses on students’ ability to construct 
meaning through text. It is an integrated, project-oriented unit through which students draw on their 
speaking, listening, writing, reading, and viewing experiences to think, learn, communicate, and 
create original work. The language arts assessment provides a variety of texts, illustrations, and 
activities intended to engage and sustain student interest in the content and sequence of assessment 
topics and tasks. In the assessment, students alternate between generating their own text and 
analyzing text provided for them. This permits students to use and enrich their literacy experiences 
as they demonstrate their knowledge of and skills in language use in varied contexts of language arts 


literacy. 


The NJASK8 uses various tasks to assess student performance, including performance-based tasks 
(speaking and writing) and multiple-choice and open-ended (reading, listening, and viewing). The 
assessment also includes audio and visual materials and formats to help students construct meaning 
as they speak, listen, write, read, and view. Finally, students use information from a reading selection 
ot selections to complete a writing project. Students are provided time to prepare notes and 


materials for their speaking presentations. 


27 


NJASK8 data are reported as scale scores ranging from 100 to 300 and are broken down into the 
same three proficiency levels as the NJASK for 6th and 7th graders: 


Advanced Proficient 250-300 
Proficient 200-249 


Partially Proficient 100-199 


The scores of students in the Partially Proficient category are considered to be below the state 


minimum for proficiency. 
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Implementation of the Targeted 
Intervention: Years 1, 2, 3, and 4 


2.A Implementation Study Design 


The extent to which treatment schools fully implemented the READ 180 curriculum was measured 
and summarized in Years 1, 2, 3 and 4 of the targeted evaluation. In Year 1, fidelity was measured 
via classroom observations and administrative data from Newark Public Schools (NPS). In Years 2, 
3, and 4 only administrative data from NPS were used to measute fidelity. In Year 5, data were not 
collected. The next few sections will detail fidelity scores for Year 4. In an effort to examine 
differences in fidelity from year to year, only fidelity subscores comparable for all four years will be 


discussed in the later sections of this report."* 


2.B Implementation Results 


To determine the degtee of fidelity to READ 180, the following components were evaluated for 
each READ 180 teacher: 


a Training 

a Class size 

a Ongoing student assessments 
= Use of instructional software 


Fidelity to the training component was measured by attendance at READ 180 trainings by teachers. 
Fidelity to the remaining three components was based on guidelines in Scholastic’s READ 180 
materials and was measured using data provided from the Scholastic Achievement Manager (SAM). 


Each of these fidelity components is discussed in more detail in the following sections. 


Classroom observation findings can be found in the Year 1 report. 
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2.B.1 Training 
2.B.1.1 Teachers 


In Year 4, new teachers were invited to attend two training days (5.5 hours each) during the school 
yeat—December 15, 2009 and March 8, 2010. Returning teachers did not attend the training days 
because they received the training in previous years. Make-up training was not offered for new 
teachers who may have missed either training day. NPS felt that the Resource Teacher Coordinators 
(RTCs) had become well versed with the READ 180 implementation and could offer the in-class 
support needed to offset the lack of additional training days. Levels of participation are outlined in 
Table 8. Table 9 provides the number and percentage of teachers at each of the levels of 


participation. 


Table 8. Participation categories for READ 180 training, Year 4 


Component Full participation Moderate participation No participation 


December and March trainings 2 days 1 day O day 


Table 9. Number and percentage of new READ 180 teachers by level of participation, Year 4 


Number % 
Full participation 2 40 
Moderate participation 3 60 
No participation 0 0 
Total 5 100 


2.B.1.2 Literacy Coaches 


In Year 4, READ 180 training was not offered to literacy coaches. Instead, READ 180 classrooms 
were heavily supported by RTCs who have had a larger role in scaffolding the implementation of the 


curriculum. 
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2.B.1.3 Other Staff 


Trainings for principals and technology coordinators were not offered in Year 4. Instead, RTCs 
continued to work closely with principals in Year 4 to ensure fidelity to the instructional model, 
analysis of reports, and using data to inform instruction. The NPS systems analyst communicated 
regularly with the technology coordinators at each treatment school to provide ongoing technical 


training and support. 


2.8.1.4 RTC In-School Coaching 


RTCs continued to support READ 180 teachers in Year 4. Treatment schools received coaching 
visits from RTCs specifically about READ 180 between September 2009 and June 2010 on an as- 
needed basis. On average, treatment schools received 19.7 visits, ranging from 14 to 28 (see Table 
10). As in Year 3, RTCs met primarily with teachers. They also met with literacy coaches, vice 
principals, and principals. Visits consisted of identifying READ 180 students, reviewing READ 180 
lesson plans, using Scholastic Reading Inventory (SRI) data and the SAM, monitoring, coaching, and 


modeling lessons. 


Table 10. Number of READ 180 RTC visits received by schools in Years 4 


Treatment School Number of Visits 
School 4 21 
School 5 16 
School 6 16 
School 8 17 
School 10 27 
School 13 20 
School 14 20 
School 15 14 
School 16 18 
School 17 28 
Average 19.7 
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2.B.2 Class Size 


Scholastic’s READ 180 materials indicate that no more than 21 students should be enrolled in a 
READ 180 classroom." The data used to measure fidelity to this component were provided by NPS 
from the SAM database. 


Sixty-six percent of READ 180 teachers teach more than one section of READ 180. Therefore, to 
determine fidelity to this component, the percentage of sections taught by the same teacher with 
fewer than 21 students is used (see Table 11). For example, if a teacher has three READ 180 
sections, and all of them are of the correct size (fewer than 21 students), then 100 percent of the 
sections meet the criteria for this component and they would be classified as fully implemented. As 


Table 12 shows, all teachers had class sizes that met READ 180 guidelines of 21 students or less. 


Table 11. Criteria for measurement of fidelity to the class size requirements of the READ 180 


curriculum 
% of sections taught 
with < 21 students Scale Level 
75-100% 4 High 
50-74% 3 Moderate-to-High 
25-49% 2 Low-to-Moderate 
0-24% 1 Low 


Table 12. Teacher-level summary scores for class size component by criteria 


Treatment school Classroom teacher Fidelity component score Fidelity level 
School 4 A 4 High 
School 4 B 4 High 
School 5 A 4 High 
School 5 B 4 High 
School 6 A 4 High 
School 6 B 4 High 
School 6 Cc 4 High 
School 8 A 4 High 
School 8 B 4 High 
School 10 A 4 High 


'S Scholastic states that “enrollment should not exceed 21 students, with 15-18 students representing an ideal class size” READ 180 Enterprise Edition 
Research Protocol and Tools — Implementation Checklist (Scholastic Research and Evaluation, 2007, p. 11). 
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Table 12. Teacher-level summary scores for class size component by criteria (continued) 


Treatment school Classroom teacher Fidelity component score Fidelity level 
School 13 A 4 High 
School 13 B 4 High 
School 13 C1/C2* 4 High 
School 13 D 4 High 
School 14 A 4 High 
School 15 A 4 High 
School 16 A 4 High 
School 17 A1/A2* 4 High 
School 17 B1/B2* 4 High 
School 17 Cc 4 High 
School 17 D 4 High 


* Teachers co-taught in these classrooms. 


2.B.3 Ongoing Student Assessment 


Scholastic’s SRI Assessment allows teachers to monitor student progress by assessing 
comprehension reading growth. This assessment tool compares both individual and group scores, 
which allows administrators to make recommendations for regrouping students based on those 
scores. Scholastic stresses the importance of ongoing monitoring so that teachers can use the 
information to most effectively differentiate instruction and check progress. Scholastic recommends 


. . . . 16 
that teachers administer a minimum of three SRI assessments per year.” 


The number of SRI assessments for all students were analyzed to determine fidelity. Criteria for 
fidelity scores are presented in Table 13 and SRI fidelity scores for each school are presented in 
Table 14. These data were provided by NPS from the SAM database. 


'6Scholastic states that regular assessment of student reading and writing proficiency is necessary through the “administration of the SRI (3-5 times per 
year)” READ 180 Enterprise Edition Research Protocol and Tools — Implementation Checklist (Scholastic Research and Evaluation, 2007, p. 11). 
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Table 13. Criteria for measurement of fidelity to the READ 180 requirements for monitoring 
student progress 


% of students with 
3 or more SRIs Scale Level 
75 - 100% 4 High 
50 - 74% 3 Moderate-to-High 
25 - 49% 2 Low-to-Moderate 
0 - 24% 1 Low 


Table 14. Teacher-level summary score for assessment component by criteria 


School Classroom teacher Fidelity component score Fidelity level 
School 4 A 4 High 
School 4 B 4 High 
School 5 A 4 High 
School 5 B 4 High 
School 6 A 4 High 
School 6 B 4 High 
School 6 Cc 4 High 
School 8 A 4 High 
School 8 B 4 High 
School 10 A 4 High 
School 13 A 4 High 
School 13 B 4 High 
School 13 C1/C2* 4 High 
School 13 D 4 High 
School 14 A 4 High 
School 15 A 4 High 
School 16 A 4 High 
School 17 A1/A2* 4 High 
School 17 B1/B2* 4 High 
School 17 Cc 4 High 
School 17 D 4 High 


* Teachers co-taught in these classrooms. 
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It should be noted that the attendance of a student could affect the number of SRIs that student 
takes, with chronically absent students or mid-year transfer students being less likely to take the full 
complement of assessments. Despite this, all teachers assessed more than 75 percent of their 
students at least three times during the school year (see Table 14). On average, teachers administered 


4.7 SRIs per student in Year 4. 


2.B.4 Instructional Software 


Part of the READ 180 instructional model consists of a 60-minute segment in which students break 


into three small groups that rotate among three stations: small group instruction, independent 
reading, and READ 180 software. 


Scholastic recommends that students use the READ 180 software a minimum of three times a week 
and 15 minutes per session.'’ The numbers of student sessions as well as the length of these sessions 
were provided by NPS from the SAM database. The percentage of students who received both a 
minimum of three sessions per week and a minimum of 15 minutes per session was used to 


determine fidelity to this component (see Table 15). 


Table 15. Criteria for measurement of fidelity to the READ 180 requirements for student 
software use 


% of students with adequate 


exposure to software Scale Level 
75 - 100% 4 High 
50 - 74% 3 Moderate-to-High 
25 - 49% 2 Low-to-Moderate 
O - 24% 1 Low 


To explore the possible reasons for the low percentage of teachers who ensured that their students 
had adequate levels of exposure, the two parts of this subscale (number of sessions and time per 


session) are examined separately in Table 16. 


As can be seen from Table 16, all teachers adhered to the recommended 15 minute length of 


session. However, fidelity to a minimum of three sessions per week appeared to be more of a 


'7§cholastic states that “to receive the full benefits of READ 180, your students should use the topic software at least 15 minutes a day” READ 180 
Enterprise Edition Placement, Assessment, and Reporting Guide (Scholastic Research and Validation, 2005, p. 81). 
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challenge. RTCs have noted instances in which students did not log off of the computer properly, 


which may have led to an underestimate in software usage. 


Table 16. Teacher-level summary score for time on instructional software by criteria 


i 
Treatment Classroom Instructional Number of eae 
school teacher minutes score _ sessions score score Fidelity level 

School 5 A 4 4 3 Moderate-to-High 
School 10 A 4 4 3 Moderate-to-High 
School 4 B 4 2 2 Low-to-Moderate 
School 5 B 4 3 2 Low-to-Moderate 
School 17 A1/A2* 4 3 2 Low-to-Moderate 
School 17 D 4 3 2 Low-to-Moderate 
School 4 A 4 1 1 Low 

School 6 A 4 1 1 Low 

School 6 B 4 2 1 Low 

School 6 C 4 2 1 Low 

School 8 A 4 2 1 Low 

School 8 B 4 3 1 Low 

School 13 A 4 3 1 Low 

School 13 B 4 1 1 Low 

School 13 C1/C2* 4 1 1 Low 

School 13 D 4 1 1 Low 

School 14 A 4 2 1 Low 

School 15 A 4 1 1 Low 

School 16 A 4 3 1 Low 

School 17 B1/B2* 4 3 1 Low 

School 17 Cc 4 3 1 Low 

Average 4.0 2.3 1.4 


* Teachers co-taught in these classrooms. 


2.B.5 Participation Summary, Year 4 


In Year 4, an overall school-level summary scale was developed to see how the different facets of 
fidelity have come together. Table 17 lists the definitions for the school-level implementation. Each 
school was assigned a level rating: Low, Low-to-Moderate, Moderate-to-High, or High (see Table 
18). Forty percent of schools achieved high implementation in all three components. The remaining 


60 percent schools had moderate-to-high implementation. 
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Table 17. Criteria for fidelity to READ 180 implementation 


Average score Level 
3.1-4 High 
2.1-3 Moderate-to-High 
1.1-2 Low-to-Moderate 
0-1 Low 


Table 18. Average schoollevel summary scores for implementation in READ 180 intervention in 


Year 4 
Ongoing t—“(<i<‘iéSOO!O!O!O!!OOCOCO#*;«CSUMmaryC” 
Treatment Professional student Instructional Average implementation 
school development* Classsize assessment _ software use score scores 

School 10 - 4 4 3 3.7 High 
School 5 - 4 4 2.5 3.5 High 
School 4 - 4 4 1.5 3.2 High 
School 17 - 4 4 1.5 3.2 High 
School 6 - 4 4 3.0 Moderate-to-High 
School 8 - 4 4 1 3.0 Moderate-to-High 
School 13 - 4 4 1 3.0 Moderate-to-High 
School 14 - 4 4 1 3.0 Moderate-to-High 
School 15 - 4 4 1 3.0 Moderate-to-High 
School 16 - 4 4 1 3.0 Moderate-to-High 


* A summary score was not calculated for professional development due to a change in the professional development model. 


2.C Barriers to Targeted Implementation, Year 4 


A serious challenge in implementation of the targeted intervention in Year 4 has been students being 
“pulled” from READ 180 for a number of reasons. Students have been pulled from their READ 
180 classrooms to attend other events in the schools (e.g., field trips, additional testing). 
Coordination and communication issues remain between the Office of Language Arts Literacy and 
the Office of Special Education. The Child Study Team continues to pose that READ 180 violates 
the Individualized Education Programs (IEPs) of some students. The district continues to ensure 


buy-in from the inclusion teachers as much as possible, but this has been difficult. 
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Another implementation challenge has been that the amount of support provided by the RTCs has 
decreased from previous years. The number of RTCs has dropped from 5 to 3 for various medical 


reasons. As a result, the number of visits varied ranging from 14 to 28. 


Teacher and principal turnover through the 4 years of the grant has been a small challenge, although 
yearly training and RTC visits keeps new and old teachers up to date on READ 180 techniques. 
However, it should be noted that there were changes and principals across all 4 years of the grant. 
Over half of the teachers in Year 4 remained in Year 3 (see Table 19). During the course of 4 years, 
new teachers have been hired, transferred, or have changed positions. Table 20 describes how many 
teachers have taught for 1 or more years of the grant. While 16 percent of teachers have remained in 


the project over the first 4 years, 46 percent of teachers have only taught for 1 out of the 4 years. 


Table 19. Teacher turnover from Year 3 to Year 4 


Year 3 teacher Year 4 teacher Number of teachers % 
Yes Yes 15 56% 
Yes No 12 44% 
No Yes 6 - 


Table 20. Number of teachers by years of participation in READ 180 


Number of years Number of teachers % 
All 4 Years 8 16 
3 Years 8 16 
2 Years 11 22 
1 Year 23 46 
Total 50 100 


The 10 treatment schools have had 19 principals during the 4 years of the project. Although there 
was little turnover between Years 3 and 4 (see Table 21), very few principals (15.8 percent) have 


remained for all 4 years of Striving Readers see Table 22. 


Table 21. Principal turnover from Year 3 to Year 4 


Year 3 principal Year 4 principal Number of principals % 
Yes Yes 9 90% 
Yes No 1 10% 
No Yes 1 - 
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Table 22. Number of principals by years in READ 180 


iWu f 7 


All 4 Years 

3 Years 4 21.1 
2 Years 4 21.1 
1 Year 8 42.1 
Total 19 100.0 


2.D Years 1-4 Implementation Summary 
2.0.1 ‘Training , Years 1-4 Summary 


At the beginning of the grant (Year 1), READ 180 training was held during the summer for all 
teachers. Make-up training was made available during the school year for those who missed the 
initial training. In Year 2, all teachers were asked to attend training as a refresher to the previous 
year’s READ 180 training. In Year 3, however, the format of training changed—only new teachers 
were invited to attend. Only one teacher, who was a new hire, attended the training. The other 
teachers new to READ 180 were unable to attend because they were brought on as staff after the 
training occurred. In Year 4, training was offered to five teachers new to READ 180. As with Year 


3, returning teachers did not attending the trainings. 


In Year 1, all of the school principals attended the implementation meeting, which outlined the 
logistics and requirements for executing READ 180 and the READ 180 training session. In Year 2, 
5 of the 10 treatment school principals attended the READ 180 training. In Years 3 and 4, principals 
did not receive training, but were updated regularly by the RTCs on READ 180 implementation. 


Few literacy coaches attended READ 180 training in Years 1 and 2. In Year 1, 20 percent of the 
coaches received training in the READ 180 curriculum. In Year 2, none of the coaches attended the 


summer training because of a scheduling conflict. In Years 3 and 4, training was not offered. 


In Years 1 and 2, all technology coordinators received training from Scholastic so that they could 
better support the installation and operation of the technology component of the curriculum. In 
Years 3 and 4, training was not offered. However, the district’s systems analyst communicated 


regularly with the technology coordinators to provide ongoing technical training and support. 
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In Year 1, data on RTC visits specific to READ 180 classrooms were not available. In Year 2, 
treatment schools recetved READ 180 coaching visits from RTCs between September 11, 2007 and 
June 24, 2008." On average, treatment schools received 19.4 visits, ranging from 7 to 38. In Year 3, 
the average visits rose to 21.1, but declined slightly to 19.7 in Year 4 (see Table 23). During these 
visits RTCs met primarily with teachers, but there were also instances in which they met with literacy 


coaches, vice principals, and principals. 


Table 23. Number of READ 180 RTC visits received by schools in Years 2, 3, and 4* 


Treatment School Year 2 Year 3 Year 4 
School 4 20 41 21 
School 5 19 22 16 
School 6 22 19 16 
School 8 38 25 17 
School 10 7 20 27 
School 13 15 30 20 
School 14 23 5 20 
School 15 18 12 14 
School 16 13 10 18 
School 17 19 27 28 
Average 19.4 21.1 19.7 


* In Year 1, visitation logs did not differentiate between visits made for the whole-school intervention and the targeted intervention; 
therefore, it was not possible to determine how many READ 180 coaching visits schools received from RTCs. 


2.D.2 Class Size, Years 1-4 Summary 


In Year 1, 74 percent of teachers had class sizes within Scholastic guidelines. In Year 2, this 
increased to 100 percent. In Year 3, 95.5 percent of teachers had class sizes that met READ 180 
guidelines of 21 students or less. In Year 4, this increased to 100 percent. Table 24 provides the 
number and percentage of teachers for Years 1, 2, 3, and 4 that were at each of the levels of fidelity 


to class size. 


18 Visits supported the whole-school intervention and the targeted intervention. 
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Table 24. Number* and percentages of teachers by level fidelity to class size requirements, 
Years 1, 2, 3, and 4 


Year 1 Year 2 Year 3 Year 4 
Number % Number % Number % Number % 

High 17 73.9 22** 100.0 21 95.5 21 100.0 
Moderate- 3 13.0 (0) 0.0 (6) 0.0 O 0.0 
to-High 
Low-to- (6) 0.0 (0) 0.0 (6) 0.0 (0) 0.0 
Moderate 
Low 3 13.0 (6) 0.0 4 4.5 (6) 0.0 

Total 23 100.0 22** 100.0 22 100.0 21 100.0 


* The number of teachers is characterized by the lead classroom teacher. In instances in which a classroom was co-taught, only the lead 
teacher was included in the teacher count. 


**Two classrooms had teachers leave their respective school mid-year. In each case, only one teacher is included in the teacher count. 


At the school-level, six schools (60 percent) recetved an average score in the High category for all 4 
years of the study. The most variation in scores occurred in Year 1, while in Year 2, all of the 


schools received an average rating in the High category. 


2.D.2.1 Changes in Class Size Fidelity Among Years 1, 2, 3, and 4 


Four schools showed an increase in their average fidelity score from Year 1 to Year 2. The majority 
of schools exhibited no change from Year 2 to Year 3. In Year 4, all schools obtained an average 
score in the High category. Interestingly, the three of the four schools that showed an increase from 
Year 1 to Year 2, maintained an average score in the High category in Year 4. Table 25 shows the 
criteria for fidelity to class size requirements, and Table 26 shows changes in class size fidelity from 


the first 3 years of the grant. 


Table 25. Criteria for fidelity to class size requirements 


Average school score Level 
3.1-4 High 
2.1-3 Moderate-to-High 
1.1-2 Low-to-Moderate 
0-1 Low 


41 


Table 26. Year 1, 2, 3, and 4 average scores by school by level of fidelity to class size 
requirements 


Change from Change from 


Treatment Year 1 Year 2 Year 3 Year 4 Year 3 to Year 1 to 
school (1-4) (1-4) (1-4) (1-4) Year 4 Year 4 
School 4 2 4 4 4 O 2 
School 5 4 4 4 4 Oo Oo 
School 6 4 4 4 4 Oo Oo 
School 8 2.5 4 2.5 4 1.5 1.5 
School 10 4 4 4 4 O O 
School 13 3.7 4 4 4 Oo 3 
School 14 4 4 4 4 0 O 
School 15 4 4 4 4 O 0 
School 16 4 4 4 4 Oo 
School 17 3.2 4 4 4 0 8 


2.D.3 Ongoing Assessment, Years 1-4 Summary 


The number of SRI assessments for all 4 years were analyzed to determine fidelity (see Table 27). 
The vast majority of teachers assessed more than 75 percent of their students at least three times 
during Year 1. In Years 2, 3, and 4, all teachers assessed more than 75 percent of their students at 


least three times during the schools year. 


Table 27. Number* and percentages of teachers by level fidelity to assessment requirements, 
Years 1, 2, 3, and 4 


Year 1 Year 2 Year 3 Year 4 
Number Number Number Number 
of of of of 

teachers % teachers % teachers % teachers % 
High 20 90.9 22 100.0 22 100.0 21 100.0 
Moderate-to-High 4 4.5 Oo 0.0 (6) 0.0 (6) 0.0 
Low-to-Moderate 1 4.5 (0) 0.0 (0) 0.0 O 0.0 
Low (6) 0.0 (6) 0.0 (6) 0.0 (6) 0.0 
Total 22** 100.0 22*** 100.0 22 100.0 21 100.0 


* The number of teachers is characterized by the lead classroom teacher. In instances in which a classroom was co-taught, only the lead 
teacher was included in the teacher count. 


**Data for one teacher was unavailable. 


***Two classrooms had teachers leave their respective school mid-year. In each case, only one teacher is included in the teacher count. 
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In terms of school-level, the average rating for all 4 years was in the High category. Table 28 defines 
each school score and Table 29 shows overall ratings of fidelity to assessment requirements across 


the 4 years of the grant. 


Table 28. Criteria for fidelity to assessment requirements 


Average school score Level 
3.1-4 High 
2.1-3 Moderate-to-High 
1.1-2 Low-to-Moderate 
0-1 Low 


Table 29. Year 1, 2, 3, and 4 average ratings by school by level of fidelity to assessment 
requirements 


Change from Change from 


Treatment Year 1 Year 2 Year 3 Year 4 Year 3 to Year 1 to 
School (1-4) (1-4) (1-4) (1-4) Year 4 Year 4 
School 4 4 4 4 4 0 Oo 
School 5 4 4 4 4 (0) (0) 
School 6 4 4 4 4 0 0 
School 8 4 4 4 4 0 Oo 
School 10 4 4 4 4 (0) Oo 
School 13 4 4 4 4 0 O 
School 14 4 4 4 4 0 O 
School 15 4 4 4 4 0 oO 
School 16 4 4 4 4 O 0 
School 17 4 4 4 4 Oo 0 


2.D.3.1 Changes in Ongoing Assessment Among Years 1, 2, 3, and 4 


There were no changes in ongoing assessment at the school level from Years 1 to 4 or between 
years. The same holds true for teachers. All teachers assessed more than 75 percent of their students 


at least three times during the school year for all 4 years of the grant. 
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2.D.4 Instructional Software, Years 1-4 Summary 


In Year 1, 65 percent of teachers ensured that more than half of their students had moderate levels 
of exposure to the instructional software. In Year 2, the percentage fell to 9 percent. In Year 3, it 
rose to 18 percent. In Year 4, only 9 percent of teachers ensured that more than half of their 


students had moderate levels of exposure to the instructional software none of the (see Table 30). 


Table 30. Number* and percentages of teachers by level fidelity to instructional software 
guidelines, Years 1, 2, 3, and 4 


Year 1 Year 2 Year 3 Year 4 
Number Number Number Number 
of of of of 

teachers % teachers % teachers % teachers % 
High 15 65.2 2 9.1 4 18.2 0 0.0 
Moderate-to-High 6 26.1 0 0.0 7 31.8 2 9.5 
Low-to-Moderate 2 8.7 2 9.1 5 22.7 4 19.1 
Low (0) 0.0 18 81.8 6 27.3 15 71.4 
Total 23 100.0 22** 100.0 22 100 21 100.0 


* The number of teachers is characterized by the lead classroom teacher. In instances in which a classroom was co-taught, only the lead 
teacher was included in the teacher count. 


**Two classrooms had teachers leave their respective school mid-year. In each case, only one teacher is included in the teacher count. 


Table 31 defines each school score and Table 32 shows overall ratings of fidelity to instructional 


software requirements across the first 3 years of the grant. 


Table 31. Criteria for fidelity to instructional software guidelines 


Average school score Level 
3.1-4 High 
2.1-3 Moderate-to-High 
1.1-2 Low-to-Moderate 
0-1 Low 
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Table 32. Year 1, 2, 3, and 4 average ratings by school by level fidelity to instructional software 


guidelines 
Change from Change from 
Treatment Year 1 Year 2 Year 3 Year 4 Year 3 to Year 1 to 
school (1-4) (1-4) (1-4) (1-4) Year 4 Year 4 
School 4 3 1 4 3 -1 0 
School 5 3.5 2 4 2.5 -1.5 -1 
School 6 3.5 1.8 4 1.5 -2.5 -2 
School 8 3.5 1 2.5 1.5 -1 -2 
School 10 4 4 4 1 -3 -3 
School 13 3.7 1 4 1 -3 -2.7 
School 14 3 1 4 1 -3 -2 
School 15 3.5 1 4 1 -3 -2.5 
School 16 3.5 1 4 1 -3 -2.5 
School 17 3.6 1.3 4 1 -3 -2.6 


In Year 1, the school-level average for the majority of schools was in the High category. In Year 2, 
however, 90 percent of schools dropped to the Low-to-Moderate category. When asked about this 
substantial drop in fidelity, NPS hypothesized that some of it may be due to instances in which 
students did not log off of the computer properly, which may have led to an underestimate in 
software usage. Fidelity to this component rebounded in Year 3 with all schools improving, 
especially 90 percent of schools in the High category. In Year 4, the majority of schools dropped 
again to the Low category. As in Year 2, instances of students improperly logging off of the 


computer may have led to an underestimate in software usage. 


2.D.4.1 Changes in Instructional Software Among Years 1, 2, 3, and 4 


Only one school remained unchanged in the High category for the first three years of the study and 
fell to the Moderate-to-High category in Year 4. There was a large decrease in fidelity for nine 
schools from Year 1 to Year 2. All of those schools increased in Year 2 and continued to increase in 
fidelity in Year 3, but decreased again in Year 4. Changes from Year 1 to Year 4 show a large 


decrease in fidelity to the component in all but one school. 
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Impacts of the Targeted 
Intervention: Years 1, 2,3 4, and 5 


3.A Study Design and Analytic Approach 


This report updates one analysis (3 years); the remainder of the impacts and implementation findings 
are for Year 4 and have been reported previously in Striving Readers Study: Targeted & Whole-School 


Interventions — Year 4. 
3.A.1 Sampling Plan 


3.A1.1 Power 


Power estimates describe how likely differences between treatment and control groups can be 
detected. Power calculations were conducted in the fall of 2006 using the following set of 


assumptions: 
1. A total of 19 schools randomly assigned to treatment and control groups 


2. About 90 students at each school participating at each time point 


3. An intraclass correlation (ICC) of .02 (2 percent of the total variation in the outcome is 
between schools) 


4. An alpha level for the statistical test set at .05 (two-tailed test) 


The power calculations assume that level 2 covariates do not explain variation in student outcomes. 
In fact, with a baseline ICC of 0.02, there is not much between-school variation to explain. Based on 
these power calculations, an effect size of .24 (Gust under a quarter of a standard deviation) can be 


detected with the number of students participating in the study. 
Using the standard deviations from the 8th grade Language Arts assessments in the Stanford 


Achievement Test 9 (SAT9) (Pearson Assessment, 1996), Table 33 illustrates the boost in scores 


because of intervention, assuming an effect size of .24. 
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Table 33. Illustrative example of the practical significance of a .24 effect 


Standard Effect size of Yearly score Total increase 
SATS subtest deviation intervention increase over 4 years 
Reading Vocabulary 46 .24 11 points 33 points 
Reading Comprehension 41 .24 10 points 30 points 
Language Arts 38 .24 9 points 27 points 


This example shows that, with an effect size of .24, the power calculation should allow a difference 
of 9 points (and higher) between treatment and control students each year on the Language Arts 
subtest to be detected. 


3.A.1.2 School Eligibility, Randomization, and Sample Size 


To participate in the Striving Readers grant, schools had to meet the following eligibility criteria: 


1, Be Title I eligible 
2. Serve a minimum of two grades (from 6, 7, 8) 
3: Not using READ 180 


4. Be categorized as “in need of improvement” under the No Child Left Behind Act 
(NCLB) 


5. Serve a minimum of 25 eligible students 


Based on these criteria, 20 schools were initially eligible. After randomization, two control schools 
later merged, leaving 19 participating schools. For the targeted portion of the grant, these schools 


were randomly assigned to either the intervention or the control condition. 


The randomization process used blocking variables. Using the number of eligible students as the 
primary blocking variable, schools were divided into three groups: large schools (more than 100 
eligible students), medium schools (51—100 eligible students), and small schools (25—50 eligible 
students). Within these three groups, schools then were sorted by number of years in need of 
improvement (INOJ) under NCLB and then by home language and special education status. Table 


34 provides details of this blocking information. 
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3.A.1.3 Student Eligibility and Sample Size 


For students to be eligible for the targeted evaluation in Year 1, they had to be enrolled in one of the 
eligible middle schools and be in grades 6, 7, or 8. Furthermore, student eligibility was based on their 
reading subtest score on the 2007 (New Jersey Assessment of Skills and Knowledge (NJASK). In 


Table 34. Blocking data used for random assignment 


No. eligible 
non-English 
No. eligible Year INOI native No. eligible Group 

School name students 05_06 language special ed Strata (1=T; O=C) 

School 17 189 Yr5 43 80 1 1 

School 7 98 Yr5 38 28 1 O b 
School 6 107 Yr5 16 37 2 1 Ma 2 
School 1 108 Yr5 0 34 2 O = 8 
School 11 182 Yr 56 86 3 0 i 
School 16 90 Yr5 3 36 4 1 

School 12 79 Yr5 1 43 4 (0) 

School 5 64 Yr5 3 37 5 1 3 
School 3 53 Yr5 12 30 5 0 1s 
School 4 80 Yr4 (0) 40 6 1 & g 
School 19 95 Yr3 O 68 6 oO 2 
School 15 69 Yr 34 26 7 1 

School 18 55 Yr 11 26 7 0 

School 10 48 Yr4 1 5 8 1 

School 2 39 Yr4 0 15 8 oO > 
School 14 37 Yr4 6 13 9 1 e g 
School 9 33 Yr3 4 5 9 0 oa 8 
School 8 27 Yr2 Oo 14 10 1 . 


New Jersey, anyone scoring below a 200 is considered “partially proficient,” which is the lowest 
category possible. Scores 200-249 are “proficient,” while scores above 249 are “advanced 
proficient.” The cut-off scores for student eligibility were set by Newark Public Schools (NPS), 
based on one standard deviation from the norm. An example of student scores and the cut-off for 
eligibility is represented graphically in Figure 4. The same student eligibility requirements are used 
for students in both treatment and control schools. They must score below the cut-off score on the 


NJASK to be included in the evaluation. The specific cut-off scores for each grade are the following: 


a 6th grade = 198 
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2 7th grade = 186 
a 8th grade = 192 


Figure 4. | Language Arts scale score frequency distribution for 19 evaluation schools (Grade 5) 


Proficiency Cut Score 
198 


te a] 135 190 265 8] 


NJASK Language Arts Scale Score 


In Years 2, 3 and 4, a second, third and forth cohort of 6th graders were added to the evaluation. 
The cut score of 198 applied to these cohorts as well. Transfer students without an NJASK score 
were not eligible to participate in the targeted evaluation. In Year 4, 1,070 students participated in 
the Striving Reader evaluation, with 577 attending treatment schools and 493 attending control 

schools. In Year 5, no new cohort of 6th grade students was added; only 8th grade students were 


assessed. 
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3.A.2 Description of the Counterfactual 


Thirty-six language arts classrooms in grades 6, 7, and 8 were observed by trained Westat researchers 
in the spring of 2007. Twenty-one of these classrooms were READ 180 classrooms, while the 


remainder were control classrooms. 


Based on these observations, the number of students per classroom varied from 3 to 25. The 
average class size was 15. Using NPS data, there is a statistically significant difference in the class 


size by treatment group (see Table 35). 


Table 35. Class size comparison, treatment versus control 


N Mean sd Sig diff? 
Control classrooms 101 18.02 5.63 * 
Treatment classrooms 43 15.70 5.09 


T-test significant at the .05 level. 


In terms of the physical environment, observers were asked to rate the classroom on a scale of 1 to 
4, with 4 being the highest possible score. As Table 36 shows, all observed classrooms scored well 
on the availability of books in the classroom and the resources displayed on the classroom walls. 
However, as might be expected, the treatment classrooms had significantly more technology-related 


resoutces available than control classrooms. 


Table 36. Physical environment of classrooms 


Treatment Control 
Mean sd Mean sd Sig diff? 
Technology 3.57 .598 2.93 .884 * 
Bulletin boards/walls (e.g., student samples 3.29 717 3.33 724 
word walls) 
Availability of books 3.52 512 3.27 799 


T-test significant at the .05 level. 


During the class period, observers also were asked to identify the literacy resources being used by 
students. Table 37 shows the results from these yes/no questions. Students in all classrooms used 
the same set of literacy resources, except in three cases: textbook use, use of computers, and use of 
audio equipment. In these cases, the treatment curriculum focuses heavily on these resources (Le., 
rBooks, instructional software, and independent reading with CD-ROMs), so there is little surprise 


that treatment classrooms would score higher in these areas. 
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Table 37. Literacy resources used in observed classrooms 


Treatment Control 
% yes sd % yes sd Sig diff? 
Reading or discussion of novels/stories/ poems 95 .229 100 000 
Textbook 79 419 21 426 * 
Articles 47 514 21 .426 
Students all read same text 74 452 77 439 
Workbook/worksheets used 76 436 73 .458 
Video/film/television 42 .507 14 .363 
Notebooks/journals 81 402 93 .258 
Computer use 95 .218 20 414 * 
Audio 80 .410 7 .258 * 


T-test significant at the .05 level. 


In terms of organization, climate, and culture, all classrooms scored very high on a 1 to 5 scale with 
5 being the highest possible score. As a safe environment for struggling readers (Le., struggling 
readers risked making mistakes, got a lot of encouragement, and read without ridicule), treatment 


classrooms scored significantly higher than control classrooms (see Table 38). 


Table 38. Classroom organization 


Treatment Control 
Mean sd Mean sd Sig diff? 
Classroom time well-structured and transitions 4.48 .190 4.33 211 
were well defined 
Participation of all students actively 4.38 .201 4.27 .248 
encouraged 
Safe environment for struggling readers 4.57 .130 4.07 .228 * 


T-test significant at the .05 level. 


Student groupings were recorded once every 10 minutes during the classroom period. Table 39 
shows the time spent in each grouping. These data show that students in treatment classrooms spent 
more time in small groups and working individually than students in control classrooms (who spent 


mote time in whole class work). 
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Table 39. Average number of occasions that the following student groupings were observed 


Treatment Control 
Number sd Number sd Sig diff? 
Whole class 2.95 1.56 4.67 2.26 * 
Small group 5.14 1.35 3.00 2.00 * 
Individual 3.38 2.91 1.53 2.39 * 


T-test significant at the .05 level. 


3.A.3 Data Collection Plan 
3.4.3.1 Student Measures 


To determine the effect of the targeted intervention on students, the scale score results of the 
Vocabulary, Reading Comprehension, and Language Arts subtests of SAT 10 were used.'” The SAT 
10 uses vertical scaling and norm-referenced scores to ensure scale scores can be directly compared 
when students are assessed with different grade-level assessments at different times, and students’ 


scores also can be compared with a larger national sample of scores on the same tests. 


The Vocabulary subtest includes concepts such as synonyms, multiple-meaning words, and use of 
context clues to decipher the meaning of unknown words. The Reading Comprehension assesses 
students’ reading achievement using text read for enjoyment (e.g., fiction, poetry), text read for 
informational or expository purposes (e.g., science, textbook material), and everyday functional text 
(e.g., directions, labels, forms). This subtest also measures students’ ability to initially understand 
explicit details in a passage, interpret information in a passage, critically analyze and evaluate 


information in a passage, and apply appropriate reading strategies. 


The Language Arts subtest is divided into three sections. The first focuses on language mechanics, 
including capitalization, punctuation, and usage. The second focuses on language expression, 
including writing strategies and sentence structure knowledge. The final section of the Language 
Arts subtest also focuses on language expression but on a higher level than the previous section. 
Students analyze written passages for the assessment of how well they recognize extraneous 


information and descriptive language and the combining of simplistic sentences. 


19 Abbreviated battery. 
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Data on the reliability of the SAT 10 are restricted to Kuder-Richardson Formula 20 (KR-20) 
internal-consistency estimates. The reliability for the abbreviated battery of Reading subtests is .89. 
The SAT 10 thus appears to hold sufficient reliability to support data inferences about the 


performance of groups of students. 


In addition to the SAT 10, the authors also estimated whether the intervention had an effect on 
student attendance. From disttict records, the authors constructed a student-level variable that was 
the number of unexcused absences from school for each student for the school year. Overall in Year 
4, students missed an average of 12.2 days of school as compared to 23.2 average missed days in 
Year 1 (see Table 40). In Year 4 there was not a significant difference between treatment and control 
schools for the number of days missed (¢472] = 1.54, p = .124). 


Table 40. Average days of school missed in Year 4 


Average number of days missed 


Overall 12.18 
Control 13.22 
Treatment 11.00 


3.A.3.2 Schedule of Data Collection in Year 5 


Data collection for Year 5 involved testing students in grade 8 from May 23 to June 10, 2011. Prior 
to data collection, field assessors attended a 1-day training program in Newark, NJ that was 
conducted by Westat. Training topics covered the study description and background, administrative 
procedures, professional conduct, confidentiality, and student testing protocols. The goals of the 


training were the following: 


7 Increase the accuracy, quality and relevance of collected data 
. Standardize data collection techniques and procedures 
: Provide explicit procedures for assessors to follow 


After training, testing materials were sent to each assessor. Approximately 2 weeks after training, 
assessors began testing all eligible students in grade 8. The initial testing occurred over a 2-week 
period. Field assessors also conducted quality assurance checks of each student’s answer sheet to 


verify completeness and demographic information and to remove stray marks before scoring by 
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Pearson Assessment (formerly Harcourt Assessment). Table 41 provides an overview of the data 


collection schedule for the spring of 2011. 


Table 41. Data collection schedule for Year 5 


Data collection activity Date 
Assessor training May 13, 2011 
Mail data collection materials to assessors May 16 - May 20, 2011 
Conduct test administration May 23 - June 10, 2011 


3.A.4 Summary of Analytic Approach 
3.A.4.1 Model Specifications 


To determine the effect of READ 180, an intent-to-treat (ITT) analysis was conducted based on 
repeated cross-sectional data, using a multilevel software package (HLM). A linear two-level model 
was used, with student and school as the two levels. Achievement for students within schools was 
predicted by a series of student and school characteristics. Student covariates were fixed across 
schools with no interactions. For the attendance outcome, a Poisson distribution was used (the 
outcome is a count of days absent during the school year). An example of HLM output is provided 


in Appendix B. 


3.A.4.2 Selection of Analytic Variables 


The student outcomes for the targeted intervention are the three reading achievement subscores 
from the SAT 10 (i.e., Reading Comprehension, Vocabulary, and Language Arts), and school 
attendance (the number of unexcused absences). Table 42 shows the covariates used in the analytic 
model. In accordance with the recommendations of the Committee for Proprietary Medicinal 
Products (2004), the authors identified all covariates prior to data collection. The categorical 
variables were dummy-coded, and all variables (except the treatment indicator) were centered on the 


grand mean. 
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Table 42. Covariates for impact analysis 


Data format Coding 
Entered at school level 

Treatment assignment Dichotomous 1=Treatment 
0=Control 

Number of eligible students Continuous 

Year in need of improvement Count 1=1st year 
2=2nd year 
3=3rd year 
4=4th year 
5=5th year 

Number of eligible English language learners (ELL) Continuous 

students 

Number of eligible Special Ed students Continuous 

Entered at student level 

Grade 6 Categorical 1=yes 
O=no 

Grade 7 Categorical =yes 
O=no 

Grade 8 Categorical 1=yes 
O=no 

Special education identification Dichotomous 1=yes 
O=no 

Free lunch eligibility Dichotomous 1=yes 
O=no 

ELL Dichotomous =yes 
O=no 

Gender Dichotomous =yes 
O=no 

African American Dichotomous 1=yes 
O=no 

Hispanic Dichotomous 1=yes 
O=no 

Baseline reading score (NJASK) Continuous 


3.A.4.3. Analysis Groups 


Students were divided into three analytic groups to examine the overall effect of 1-3 years of the 
intervention (see Table 43). The first analytic group includes all students who were given the 
opportunity to receive 1 year of intervention. The second group includes all students who were 
given the opportunity to receive 2 years of treatment. Finally, the third group includes students who 


were given the opportunity to receive 3 years of intervention. 
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Table 43. Analysis groups by year and grade 


Year 1 


Year 2 


Year 5 


6th 
grade 


7th 
grade 


7th 
grade 


8th 
grade 


8th 
grade 


8th 
grade 


8th 
grade 


(1) Availability of 1 year of 
intervention for 6th, 7th, 
& 8th graders 
N = 2555 


v 


(2 


~~ 


Availability of 2 years of 
intervention for 7th & 
8th graders 

N =1520 


(3) Availability of 3 years of 
intervention for 8th 
graders 
N = 1023 


3.A.4.4 Missing Data 


Some data were missing for one of the covariates listed in Table 42: free and reduced lunch status. 
However, the amount of missing data was minimal, so no imputation was conducted. Some data 
also were missing for student outcomes as shown in Table 44.”” Non-random missing data could 
compromise the internal validity of the evaluation if it led to the treatment and control groups no 
longer being comparable. Overall, 90 percent of those 1,070 eligible students in Year 4 took the SAT 
10 (441 control students and 518 treatment students). Year 4 students who did not have SAT 10 
results were compared to those students with SAT 10 scores, and no statistically significant 
difference was found on their baseline NJASK scores (499] = .32, p = .75). The lack of a difference 
increases confidence that missing data has not compromised treatment and control group 


comparability. 


Table 44. Missing data for student outcomes, by analytic group 


Total 
number Number of 
Analytic group Outcome variables tested missing Missing % 
(1) Availability of 1 year of treatment for Comprehension 2555 202 7.91 
6th, 7th, & 8th graders (combined) Vocabulary 2555 298 11.66 
Language Arts 2555 336 13.15 
Attendance 2555 255 9.98 
(2) Availability of 2 years of treatment for Comprehension 1520 102 6.71 
7th & 8th graders Vocabulary 1520 113 7.43 
Language Arts 1520 120 7.89 
Attendance 1520 416 27.37 
(3) Availability of 3 years of treatment for Comprehension | —»—-1023 89 8.70 
Sth graders Vocabulary 1023 89 8.70 
Language Arts 1023 92 8.99 
Attendance 1023 303 29.62 


20 Consistent with the analysis plan, the authors did not impute missing data for outcome variables. 
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3.A.4.5 Subgroup Analyses 


To examine the effect of treatment on specific subpopulations of students, students in each analytic 


group were divided into the following five subgroups: 


i Female students 

2: Male students 

3, African Ametican students 
4, Hispanic students 


5. Special education students. 


3.B Description of the 1st, 2nd, 3rd, 4th, and 5th Year Samples 
3.B.1 Characteristics of Schools and Students 

3.B.1.1 Schools 

Nineteen middle schools were selected for the targeted intervention in Year 1. Of these schools, 10 


were tandomly assigned to recetve READ 180, and 9 were randomly assigned to the control 


condition. Randomization was maintained in Year 5 and all schools remained part of the sample. 


3.B.1.2 Students 
Figure 5 shows the evolution of the student sample throughout the 5 years of the project. 


Nonrandom attrition of individuals from randomly assigned groups can cause the groups to be no 
longer comparable and can have the same effects as self-selection bias, but during the experiment 
rather than before it. The What Works Clearinghouse (WWC) (2008) has established benchmarks 
for tolerance levels of attrition bias. Attrition rates have been examined for each of the three analytic 
groups, and the results are presented in Table 45. The overall attrition rates for the three analytic 


groups range from 6.8 percent to 16.6 percent. Differential attrition rates between the treatment and 
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Figure 5. Flow of students in the targeted intervention 


YEAR 1 


Treatment 
n=708 


YEAR 2 


Added 6th Grade Sample 
n=247 


Treatment 
n=648 


Added to sample n=26 
Incoming transfers n=26 


Eligible students 
N=1,617 


Exclude from sample n=272 


Transfers within district n=96 


Transfers out of district n=176 


Initial Sample 
n=1,371 


Schools Randomized 


Excluded Ineligible 
n=6 


Excluded 9th Grade Sample 
Treatment n=238 
Control n=192 


Transfers Out of District 
Treatment n=70 
Control n=91 


Control 
n=663 


Added 6th Grade Sample 
n=210 


Continued on next page 
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Figure 5. Flow of students in the targeted intervention (continued) 


YEAR 3 


Treatment Control 
n=648 n=584 


Added 6th Grade Sample Added 6th Grade Sample 
n=276 n=244 


Excluded Ineligible 
n=51 


Excluded 9th Grade Sample 
Treatment n=190 
Control n=190 


Transfers Out of District 
Treatment n=66 
Control n=80 


Transfers into District Transfers into District 
n=0 n=6 


Treatment Control 
n=638 n=529 


YEAR 4 


Added 6th Grade Sample Added 6th Grade Sample 
n=210 n=187 


Excluded Ineligible 
n=1 


Excluded 9th Grade Sample 
Treatment n=206 
Control n=189 


Transfers Out of District 
Treatment n=64 
Control n=34 


Treatment 
n=577 


Continued on next page 
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Figure 5. Flow of students in the targeted intervention (continued) 


YEAR 5 


Treatment Control 
n=577 n=493 


Added 6th Grade Sample Added 6th Grade Sample 
n=0 n=0 


Excluded Ineligible 
n=7 


Excluded 9th Grade Sample 
Treatment n=178 
Control n=147 


Transfers Out of District 
Treatment n=25 
Control n=11 


Excluded 7th Grade Sample 
Treatment n=202 
Control n=186 


Transfers into District Transfers into District 
n=0 n=0 


Treatment Control 
n=170 n= 144 


* The numbers in this report differ from the NPS annual performance reports. For evaluation purposes, Westat considers students as 
they were originally assigned to a condition while NPS reports students’ current school. 
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Table 45. Overall and differential attrition rates by analytic group 


Analytic Sample Attrition Rate 


Overall Differential 
Analytic Group Treatment Control attrition attrition 

(1) Availability of 1 year of treatment for 

6th, 7th, & 8th graders 6.3% 7.3% 6.8% 1.1% 
(2) Availability of 2 years of treatment for 

7th & 8th graders 14.6% 18.3% 16.6% 3.7% 
(3) Availability of 3 years of treatment for 

8th graders 14.4% 15.0% 22.6% 0.7% 


control groups were all below 4 percent. None of the differences in attrition rates between the 
treatment and control groups are statistically significant. These overall and differential attrition rates 


are within the acceptable range specified by WWC standards. 


The demographics of the 314 students eligible for the targeted intervention in Year 5 evaluation are 
similar to that of the students in NPS as a whole in several respects. Most of the Striving Readers 
students are African American (54 percent) or Hispanic (46 percent), compared to 55 and 43 percent 
in NPS as a whole. Table 46 provides detailed demographic data for Year 5 students in the treatment 


and comparison groups. 


Table 46. Characteristics of Year 5 students in the targeted intervention, by treatment status 


Students in Students in 
Number (column %) treatment schools control schools All students 
Total number of 8" grade students 170 144 314 
(54%) (46%) 
Average no. of 8 grade students per school 17.0 16.0 16.5 
Gender 
Male 66% 65% 65% 
Female 34% 35% 35% 
Status 
Economically disadvantaged 91% 83% 88% 
ELLs 12% 11% 12% 
Special education 42% 37% 39% 
Race/ethnicity 
African American 49% 53% 51% 
Hispanic 46% 45% 45% 
Caucasian <1% - <1% 
Other 1% 2% 2% 
Not identified 3% - 2% 
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3.B.2 Tests of Equivalence for Treatment and Control Schools 


Equivalence between treatment and control schools was tested in Year 1. Of the seven baseline 
variables tested for balance, one variable demonstrated a significant difference between treatment 
and control groups. As shown in Table 47, treatment schools had significantly more students eligible 
for free and reduced price lunch than control schools. However, this variable was incorporated into 


the analysis model as a student-level covariate and should not influence the impact estimates. 


Table 47. Balance test for treatment and control groups, Year 1 


Control Treatment 
Variable Mean Mean DF t Value Pr > |t| 
Males 54.75% 57.20% 1368 -0.91 0.361 
Eligible free/reduced lunch 83.86% 91.38% 1368 -4.27 <.0001* 
ELL 3.34% 2.07% 1368 0.92 0.357 
Special education student 29.08% 28.33% 1368 0.28 0.776 
Rec’d supplemental reading instruct 20.42% 21.49% 1368 -0.57 0.570 
African American 71.13% 70.07% 1368 0.40 0.688 
Baseline state assessment score 176.63 177.28 1368 -0.74 0.458 


Because the addition of new cohorts in Years 2, 3 and 4, balance tests were repeated for each year. 
In Year 2, none of the eight variables tested demonstrated a significant difference between treatment 
and control groups. However, the balance tests run on the Year 3 analytic groups indicated 
significant differences between the treatment and control groups on the percentage of students 
eligible for free/reduced-price lunch in Analytic Group 1. No significant differences were found 


between the treatment and control groups in the Year 4 or Year 5 sample. 


3.C Effects on Students 


Effects on students in each of the three analytic groups are presented in this section. Two aspects of 
the results are discussed for each group: whether any of the results ate statistically significant at the 
.05 level, and whether any of the results reach an effect size threshold of .20. It has been noted that 
when considering the practical importance of effect sizes, the context of the type of outcome being 
measured and the sample being studied should be taken into account (Hill, Bloom, Black, & Lipsey, 
2008). Effect sizes were calculated using Glass’s A (Rosenthal, 1994) and represent a change in 


standard deviation due to being part of the treatment condition. For example, an effect size of .25 
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indicates that average scores for students in the treatment group are a quarter of a standard deviation 
higher than students’ scores in the control group (see Appendix C for a table of standard deviations 


used to calculate effect sizes). 


Following the summary of the findings, tables are provided that include the means, effect sizes, and 
p-values for treatment and control groups. Furthermore, Appendix C includes detailed tables of 
model results. The Bloom “No-Show”™ rates were calculated for each analytic group and were 
found to be quite low, between 1 and 3 percent. As a result, the Bloom adjustment was not used to 


convert the ITT estimates to treatment-on-treated (TOT) impact estimates. 


3.C.1 Analytic Group 1 


Analytic Group 1 includes the 6th, 7th, and 8th graders from the Year 1 sample; and the additional 
cohorts of 6th graders from subsequent years. The goal of this analytic group was to determine if 

treatment students outperformed students in the control group after 1 year of potential exposure to 
READ 180. All grades were combined to provide the largest possible sample size, thus maximizing 


power. 


No significant effects were found for this group as a whole. Moreover, effect sizes were small (see 
Table 48). 


Table 48. Analytic Group 1 Overall—effect of READ 180 after 1 year 


Regression-adjusted 
Unadjusted means (SD) means Estimated Effect 
Outcome Control Treatment Control Treatment impact size p-value 

Attendance 21.74 23.51 

(20.80) (21.52) 19.27 19.33 0.06 0.00 0.443 
Vocabulary 611.77 615.37 

(31.65) (29.98) 613.37 614.76 1.39 0.04 0.319 
Comprehension 608.46 610.51 

(27.98) (27.95) 609.11 610.24 1.13 0.04 0.344 
Language Arts 597.88 598.66 

(26.58) (24.91) 598.40 599.10 0.70 0.03 0.397 
Number of students 1205 1350 
Number of schools 9 10 


21 “A situation in which some persons randomly assigned to the program do not participate, but follow-up data are available for these persons as well 
as for all participants and control group members” (Bloom, 1984, p. 226). 
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Subgroup analyses were then performed on this analytic group. Significant treatment effects were 


found for special education students on the Vocabulary subtest (effect size = 0.13, p=0.047). Tables 


49-53 provide the results of the subgroup analyses for Analytic Group 1. 


Table 49. Analytic Group 1 Females—effect of READ 180 after 1 year 


Regression-adjusted 
Unadjusted means (SD) means Estimated Effect 
Outcome Control Treatment Control Treatment effect size p-value 

Attendance 21.84 24.16 

(20.93) (22.34) 18.50 18.60 0.10 0.00 0.348 
Vocabulary 609.93 615.10 

(30.94) (30.53) 612.44 613.77 1.33 0.04 0.609 
Comprehension 610.70 613.90 

(25.99) (28.19) 612.05 614.00 1.94 0.07 0.190 
Language Arts 602.72 604.14 

(26.20) (24.88) 603.00 605.11 2.10 0.08 0.137 
Number of students 512 563 
Number of schools 9 10 
Table 50. Analytic Group 1 Males—effect of READ 180 after 1 year 

Regression-adjusted 
Unadjusted means (SD) means Estimated Effect 
Outcome Control Treatment Control Treatment effect size p-value 

Attendance 21.66 23.02 

(20.72) (20.91) 19.90 19.92 0.02 0.00 0.770 
Vocabulary 613.15 615.61 

(32.13) (29.59) 613.60 615.91 2.32 0.07 0.089 
Comprehension 606.75 608.07 

(29.30) (27.56) 606.83 607.93 1.10 0.04 0.462 
Language Arts 594.26 594.77 

(26.31) (24.22) 594.96 595.12 0.16 0.01 0.902 
Number of students 693 786 
Number of schools 9 10 
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Table 51. Analytic Group 1 African American—effect of READ 180 after 1 year 


Regression-adjusted 
Unadjusted means (SD) means Estimated Effect 
Outcome Control Treatment Control Treatment effect size p-value 

Attendance 21.40 24.05 

(20.45) (22.41) 19.40 19.51 0.14 0.01 0.162 
Vocabulary 612.43 616.32 

(32.42) (30.16) 614.22 615.52 1.30 0.04 0.492 
Comprehension 607.97 610.68 

(27.23) (27.86) 607.77 610.26 2.48 0.09 0.287 
Language Arts 597.02 599.53 

(25.60) (25.03) 597.63 599.35 1.73 0.07 0.160 
Number of students 705 740 
Number of schools 9 10 
Table 52. Analytic Group 1 Hispanic—effect of READ 180 after 1 year 

Regression-adjusted 
Unadjusted means (SD) means Estimated Effect 
Outcome Control Treatment Control Treatment effect size p-value 

Attendance 22.09 23.38 

(21.24) (20.55) 15.76 16.05 0.29 0.01 0.040* 
Vocabulary 611.18 614.31 

(30.82) (29.71) 612.51 615.33 2.81 0.09 0.154 
Comprehension 609.46 610.47 

(29.13) (28.20) 611.53 612.64 1.10 0.04 0.510 
Language Arts 599.05 598.11 

(28.11) (24.94) 599.61 599.36 -0.25 -0.01 0.833 
Number of students 485 576 
Number of schools 9 10 


* Significant at 0.05 level. 
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Table 53. Analytic Group 1 Special Education—effect of READ 180 after 1 year 


Regression-adjusted 


Unadjusted means (SD) means Estimated Effect 
Outcome Control Treatment Control Treatment effect size p-value 

Attendance 20.80 22.90 

(20.06) (19.81) 19.48 19.52 0.04 0.00 0.685 
Vocabulary 603.45 608.34 

(31.83) (32.17) 604.11 608.13 4.03 0.13 0.047* 
Comprehension 600.64 601.51 

(27.05) (27.36) 600.19 601.51 1.32 0.05 0.417 
Language Arts 588.85 589.31 

(23.92) (22.31) 588.63 589.69 1.06 0.04 0.459 
Number of students 486 543 
Number of schools 9 10 


* Significant at 0.05 level. 


Table 54 summarizes statistically significant findings and effect sizes for Analytic Group 1. Special 


education students in the treatment group had significant higher Vocabulary subtest scores than 


their counterparts in the control group. Hispanic students in the treatment group had more absent 


days compared to the Hispanic students in the control group. However, the difference is negligible 


since the effect size is only 0.01. For the rest of the subgroups, READ 180 program had no impact 


on students after 1 year. 


Table 54. Analytic Group 1 Summary—effect of READ 180 after 1 year 


Vocabulary Comprehension Language Arts Attendance 

Subgroup ES Sig ES Sig ES Sig ES Sig 
Overall 
Female 
Male 
African American 
Hispanic 0.01 * 
Special education 0.13 * 


3.C.2 Analytic Group 2 


Analytic Group 2 includes the 7th and 8th graders in Year 2 (who were 6th and 7th graders in Year 
1), the 7th graders in Year 3 (who were 6th graders in Year 2), and the 7th graders in Year 4. This 


group combines all students who had 2 years of potential exposure to the intervention. Analytic 
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Group 2 showed overall effects in Comprehension (see Table 55). However, the effect size of this 


finding was relatively small, less than the .20 threshold. 


Table 55. Analytic Group 2 Overall—effect of 2 years of READ 180 


Regression-adjusted 
Unadjusted means (SD) means Estimated Effect 
Outcome Control Treatment Control Treatment effect size p-value 

Attendance 23.06 22.17 

(22.44) (20.22) 19.09 19.14 0.05 0.00 0.515 
Vocabulary 629.48 629.06 

(27.16) (26.82) 628.20 629.83 1.63 0.06 0.183 
Comprehension 622.58 623.06 

(26.24) (25.33) 620.85 624.44 3.59 0.14 0.019* 
Language Arts 611.28 609.56 

(25.66) (24.64) 609.12 611.23 2.11 0.08 0.295 
Number of students 706 814 
Number of schools 9 10 


* Significant at 0.05 level. 


In subgroup analysis, significant treatment effects were found in female students’ attendance (effect 
size = 0.01, p= 0.01), male students’ Comprehension scores (effect size = 0.21, p= 0.004), African 
American students’ Comprehension scores (effect size = 0.15, p=0.045), and special education 


students’ Comprehension scores (effect size = 0.22, p=0.007). 


Female students who participated in READ 180 had more absent days than female students in the 
control group. However, the different is very small with an effect size of 0.01. Special education, 
African American, and male students apparently have benefited from 2 years of exposure to READ 
180 as demonstrated by their scores in Comprehension. Tables 56 — 60 present the full subgroup 


results. 
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Table 56. 


Analytic Group 2 Female—effect of 2 years of READ 180 


Regression-adjusted 


Unadjusted means (SD) means Estimated Effect 
Outcome Control Treatment Control Treatment effect size p-value 

Attendance 21.86 23.60 

(21.19) (19.14) 17.12 17.33 0.21 0.01 0.010* 
Vocabulary 630.49 629.75 

(26.04) (23.55) 630.00 630.63 0.64 0.02 0.473 
Comprehension 626.85 625.63 

(26.95) (25.07) 625.73 626.81 1.08 0.04 0.469 
Language Arts 616.83 615.06 

(25.96) (25.35) 615.12 616.60 1.48 0.06 0.327 
Number of students 325 340 
Number of schools 9 10 
* Significant at 0.05 level. 
Table 57. Analytic Group 2 Male—effect of 2 years of READ 180 

Regression-adjusted 
Unadjusted means (SD) means Estimated Effect 
Outcome Control Treatment Control Treatment effect size pvalue 

Attendance 24.13 21.06 

(23.47) (20.99) 20.37 20.30 -0.07 0.00 0.452 
Vocabulary 628.61 628.57 

(28.09) (28.94) 626.69 629.57 2.88 0.10 0.185 
Comprehension 618.94 621.25 

(25.09) (25.39) 617.19 622.40 5.21 0.21 0.004* 
Language Arts 606.52 605.68 

(24.44) (23.38) 604.59 607.02 2.43 0.10 0.328 
Number of students 381 473 
Number of schools 9 10 


* Significant at 0.05 level. 
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Table 58. Analytic Group 2 African American—effect of 2 years of READ 180 


Regression-adjusted 
Unadjusted means (SD) means Estimated Effect 
Outcome Control Treatment Control Treatment effect size p-value 

Attendance 22.37 22.21 

(22.23) (20.67) 20.63 20.59 -0.04 0.00 0.713 
Vocabulary 628.93 630.82 

(26.86) (26.35) 629.77 631.07 1.30 0.05 0.329 
Comprehension 621.05 624.88 

(26.59) (24.43) 621.30 625.28 3.99 0.15 0.045* 
Language Arts 608.80 610.38 

(25.01) (23.19) 608.82 611.09 2.26 0.09 0.333 
Number of students 396 431 
Number of schools 9 10 
*Significant at 0.05 level. 
Table 59. Analytic Group 2 Hispanic—effect of 2 years of READ 180 

Regression-adjusted 
Unadjusted means (SD) means Estimated Effect 
Outcome Control Treatment Control Treatment effect size p-value 

Attendance 24.27 22.48 

(23.13) (19.90) 18.52 18.68 0.16 0.01 0.192 
Vocabulary 630.36 627.14 

(27.85) (27.48) 625.89 630.89 5.00 0.18 0.216 
Comprehension 624.48 621.25 

(25.92) (25.83) 621.54 623.43 1.90 0.07 0.341 
Language Arts 614.54 609.29 

(26.49) (26.38) 609.28 612.77 3.49 0.13 0.064 
Number of students 296 361 
Number of schools 7 10 
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Table 60. Analytic Group 2 Special Education—effect of 2 years of READ 180 


Regression-adjusted 


Unadjusted means (SD means 
i 2) Estimated Effect 
Outcome Control Treatment Control Treatment effect size p-value 

Attendance 23.72 22.97 

(23.78) (20.99) 20.00 20.05 0.05 0.00 0.607 
Vocabulary 621.31 621.49 

(26.78) (26.60) 619.98 622.22 2.24 0.08 0.240 
Comprehension 614.02 615.25 

(25.24) (24.59) 611.20 616.72 5.52 0.22 0.007* 
Language Arts 601.40 600.46 

(24.80) (23.18) 600.37 602.12 1.76 0.07 0.504 
Number of students 283 361 
Number of schools 9 10 


* Significant at 0.05 level. 


Table 61 summarizes statistically significant findings and effect sizes greater than 0.20 for Analytic 


Group 2. The READ 180 curriculum significantly increased the average male, African American and 


special education students’ Comprehension subtests scores. 


Table 61. Analytic Group 2 Summary—effect of 2 years of READ 180 


Vocabulary Comprehension Language Arts Attendance 
Subgroup ES Sig ES Sig ES Sig ES Sig 

Overall 0.14 * 

Female 0.01 * 
Male 0.21 * 

African American 0.15 . 

Hispanic 

Special education 0.22 * 


* Significant at 0.05 level. 


3.C.3 Analytic Group 3 


Analytic Group 3 is comprised of the 8th graders in Year 3 (who were Oth graders in Year 1) and 8th 


graders in Year 4 (who were 6th graders in Year 2) and 8th graders in Year 5 (who were 6th graders 


in Year 3). The goal of this analysis group was to determine if treatment students who had 3 years of 


Te 


READ 180 outperformed students in the control group. No significant differences between 


treatment and control group students were found (see Table 62). 


Table 62. Analytic Group 3 Overall—effect of 3 years of READ 180 


Regression-adjusted 
Unadjusted means (SD) means Estimated Effect 
Outcome Control Treatment Control Treatment effect size p-value 

Attendance 15.04 14.71 

(15.30) (11.55) 15.83 15.63 -0.19 -0.01 0.080 
Vocabulary 642.97 642.29 

(28.21) (25.95) 641.47 642.91 1.44 0.05 0.509 
Comprehension 641.17 641.47 

(23.91) (22.83) 640.33 641.74 1.41 0.06 0.398 
Language Arts 621.53 622.91 

(22.63) (24.11) 621.48 623.15 1.67 0.07 0.320 
Number of students 471 552 
Number of schools 9 10 


When subgroup differences were examined, males (effect size = -0.02, p=0.037) and special 
education students (effect size = -0.02, p=0.001) in the treatment group were found to have better 
school attendance compared to the corresponding students in the control groups. Tables 63 — 67 


present the full subgroup results. 


Table 63. Analytic Group 3 Female—effect of READ 180 


Regression-adjusted 
Unadjusted means (SD) means Estimated Effect 
Outcome Control Treatment Control Treatment effect size p-value 

Attendance 13.66 14.88 

(11.94) (11.86) 13.84 13.80 -0.04 0.00 0.675 
Vocabulary 642.06 643.54 

(23.81) (23.99) 641.87 642.11 0.24 0.01 0.922 
Comprehension 643.15 643.11 

(23.08) (23.47) 643.80 642.36 -1.44 -0.06 0.504 
Language Arts 626.08 629.28 

(22.10) (25.52) 626.67 627.32 0.65 0.03 0.786 
Number of students 211 225 
Number of schools 9 10 
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Table 64. Analytic Group 3 Male—effect of READ 180 


Regression-adjusted 
Unadjusted means (SD) means Estimated Effect 
Outcome Control Treatment Control Treatment effect size p-value 

Attendance 16.23 14.58 

(17.62) (11.34) 17.34 17.03 -0.31 -0.02 0.037* 
Vocabulary 643.74 641.46 

(31.47) (27.19) 641.01 643.75 2.74 0.09 0.342 
Comprehension 639.49 640.37 

(24.51) (22.37) 638.07 641.26 3.19 0.13 0.133 
Language Arts 617.72 618.67 

(22.40) (22.17) 617.57 619.88 2.31 0.10 0.189 
Number of students 260 327 
Number of schools 9 10 
* Significant at 0.05 level. 
Table 65. Analytic Group 3 African American—effect of READ 180 

Regression-adjusted 
Unadjusted means (SD) means Estimated Effect 
Outcome Control Treatment Control Treatment effect size p-value 

Attendance 15.54 14.41 

(17.16) (12.65) 17.93 17.68 -0.25 -0.01 0.114 
Vocabulary 640.79 642.90 

(29.60) (25.09) 640.49 641.95 1.46 0.05 0.594 
Comprehension 638.59 641.34 

(25.10) (24.06) 638.14 640.80 2.66 0.11 0.281 
Language Arts 620.69 624.08 

(21.78) (24.24) 620.64 623.17 2.53 0.12 0.167 
Number of students 263 287 
Number of schools 9 10 
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Table 66. Analytic Group 3 Hispanic—effect of READ 180 


Regression-adjusted 


Unadjusted means (SD) means Estimated Effect 
Outcome Control Treatment Control Treatment effect size p-value 
Attendance 14.46 15.26 
(12.88) (10.42) 15.44 15.33 -0.11 -0.01 0.481 
Vocabulary 645.59 641.57 
(26.22) (27.21) 646.60 645.86 -0.73 -0.03 0.892 
Comprehension 644.06 641.90 
(22.07) (21.78) 643.60 644.80 1.20 0.05 0.632 
Language Arts 622.40 622.08 
(23.50) (23.92) 625.32 626.09 0.77 0.03 0.661 
Number of students 199 248 
Number of schools 6 9 
Table 67. Analytic Group 3 Special Education—effect of READ 180 
Regression-adjusted 
Unadjusted means (SD) means Estimated Effect 
Outcome Control Treatment Control Treatment effect size p-value 
Attendance 18.01 15.67 
(20.02) (13.08) 21.39 20.92 -0.46 -0.02 0.001* 
Vocabulary 634.03 635.56 
(26.86) (25.09) 635.89 634.58 -1.31 -0.05 0.587 
Comprehension 634.75 635.15 
(24.30) (22.96) 635.19 634.48 -0.71 -0.03 0.677 
Language Arts 612.28 615.85 
(20.00) (21.29) 613.22 616.30 3.09 0.15 0.058 
Number of students 203 257 
Number of schools 9 10 


Table 68 summarizes statistically significant findings and effect sizes for Analytic Group 3. Male and 


special education students in the treatment groups had fewer absent days than those students in the 


control groups after participating in READ 180 program for 3 years. 
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Table 68. Analytic Group 3 Summary—effect of READ 180 


Vocabulary Comprehension Language Arts Attendance 

Subgroup ES Sig ES Sig ES Sig ES Sig 
Overall 
Female 
Male -0.02* 
African American 
Hispanic 
Special education -0.02 * 


* Significant at 0.05 level. 


3.D Summary and Discussion 


3.D.1 Overall Findings 


Our study indicates that 1 year of exposure to the READ 180 treatment yielded no significant 
impact on student literacy performance and their school attendance. However, after 2 years of 
program exposure, significant effect was found in students’ reading comprehension. However, this 
impact was not sustained when we examined those students with 3 years of program exposure. The 
lack of continued impact could be due to a number of different factors, including teacher training, 
program implementation fidelity, and student motivation. During Year 4, READ 180 training was 
not offered to continuing teachers in any of the treatment schools. Moreover, implementation 
fidelity data indicate that in Year 4, student use of the READ 180 software was below the dosage 
recommended by the developer. Lastly, after 3 years of using READ 180, a curriculum designed to 
be implemented over 2 years, student motivation may have suffered from the pacing of the 


instruction. 


3.D.2 Subgroup Findings 


It is important to also consider the significant effects found when looking at specific subgroups of 
students. READ 180 appeared to be particularly effective for special education students. Special 
education students with 1 year of exposure to READ 180 scored significantly higher on the 
Vocabulary subtest than control students. Special education students with 2 years of treatment 


scored higher than control students on the Comprehension subtest. However this improvement was 
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not sustained for students with 3 years of exposure to the READ 180 program. On the other hand, 
special education students with 3 years of exposure had better school attendance compared to 


special education students in the control group. 


These findings indicate that students with more complex educational needs have benefited from 
participating in the READ 180 program. We hypothesize that, since technology has been shown to 
play a role in helping special education students improve their literacy skills (Anderson, Anderson & 
Cherup, 2009), the READ 180 software with its focus on individualized instruction could be the 
driving force behind the impacts for special education students. In addition, small group instruction 
has been found to be important for special education students (McDonnell et al., 2006). READ 180 
also focuses on providing literacy instruction in small group settings, which could specifically benefit 


students with special needs. 


Another subgroup that appeared to benefit from READ 180 was males. Male students with 2 years 
of exposure scored significantly higher on the Comprehension subtest; this significant finding also 
had substantial effect sizes (0.21). However, male students with 3 years of exposure did not 
demonstrate continued improvement in their reading comprehension. Although these subtest results 
did not show improvement, male students with 3 years of exposure had fewer absent days than male 


students in the control. 


African American students also showed positive effects from exposure to READ 180. After 2 years 
of program participation, African American students in the treatment group scored significantly 
higher in Comprehension subtest than the control group. However, this effect was not found after 3 


years of exposute. 


Among Hispanic students, few significant effects were observed. Hispanic students with 2 years of 
READ 180 exposure scored significantly higher on the Language Arts subtest than the control 
group. None of the other findings were either statistically or practically significant, including effects 


after 3 years of exposure to the program. 


The finding that the READ 180 curriculum has a positive effect on the literacy skills of male and 
special education students is promising because these two subgroups of students tend to have low 
literacy achievement (U.S. Department of Education, 2010) and are most in need of effective literacy 
interventions. Data from the NAEP assessment indicate that there are literacy achievement gaps 
based on both race and gender (U.S. Department of Education, 2010). In addition, compared to 


regular education students, 30-40 percent fewer special education students meet the criteria for 
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proficiency on state reading assessments (Center on Education Policy, 2009). The findings from this 
report contribute to the evidence supporting READ 180 as an effective intervention for 
strengthening the literacy skills of students from subgroups that are at risk of low literacy 


achievement. 


3.D.3 Benjamini-Hochberg Multiple Test Adjustment 


Due to the number of tests being run, false discoveries ate a concern. Therefore, we subjected all of 
the findings to Benjamini-Hochberg multiple testing correction (1995). After the adjustment, none 


of the overall and subgroup differences remained statistically significant (See Table 69). 


Table 69. Summary of analysis findings by subgroups 


African Special 
Analysis Overall Female Male American Hispanic _ education 
groups Outcomes ES Sig ES Sig ES Sig ES Sig ES Sig ES_ Sig 
1.1 yearof Attendance 
exposure Vocabulary v 
Comprehension 
Language Arts 
2. 2 years Attendance 
of exposure Vocabulary 
Comprehension v v v v v 
Language Arts 
3.3 years Attendance v v 
of exposure Vocabulary 


Comprehension 


Language Arts 


¥ Positive at either p<.05 or effect size >.20 before adjustment. Only attendance for Year 3 special education students remained positive 
after adjustment. 
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Implementation of the Whole-School 4. 
Intervention: Years 1, 2, 3, and 4 


4.A Implementation Study Design 


Professional development is widely used as a method for training teachers in appropriate methods of 
instruction, and research indicates it may produce changes in teachers’ instructional behaviors over 
time (Landry, Anthony, Swank, & Monseque-Bailey, 2009; Onchwari & Keengwe, 2008; Wallace, 
2009). Professional development can be delivered in various formats, including large-group 
instruction as well as classroom embedded, individual instruction. Recent literature suggests a 
combination approach (large-group with individual coaching) may be particularly effective at 
influencing teacher practices (Cantrell, Burns, & Calloway, 2009). Newark Public Schools (NPS) 
incorporated both large-group professional development and classroom-embedded or “in-school” 
coaching into the design of its whole-school intervention. Two developers, New Jersey City 
University (NJCU) and National Urban Alliance (NUA), delivered professional development to all 
6th, 7th, and 8th grade teachers in the 19 Striving Readers schools. NJCU trainings were available to 
language arts teachers and NUA trainings were available to content teachers. Teachers who taught 
both language arts and content areas were eligible to participate in both NJCU and NUA trainings. 
Both NJCU and NUA paired large-group professional development sessions with in-school, 


classroom-embedded sessions. 


The extent to which teachers in Striving Readers schools fully participated in the whole-school 
intervention was measured and summarized in the 4 years of the evaluation. Fidelity was measured 
by obtaining records of teachers’ participation in NJCU and NUA large-group trainings, and receipt 


of in-school coaching visits. 


This chapter begins with a description of the developers’ roles in implementing the professional 
development and the curriculum used by each developer. Next, it presents detailed implementation 
findings for NJCU and NUA teachers in Year 4. Finally, it presents a summary of implementation in 
Years 1 through 4 along with a description of changes in implementation across the first 4 years of 


the grant. 
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4.A.1 Developers’ Role in Implementation 


The initial role of the developers was to provide professional development to teachers, coaches, 
Resource Teacher Coordinators (RTCs), and building administrators through the summer institutes; 
large-group sessions during the school year; and ongoing in-school coaching visits. As the project 
evolved over time the developers engaged in more in-school coaching and less on large-group 
training, and focused training more for teachers, not other staff. Each professional development 


curriculum is described in detail in the following sections. 


The role of NJCU in Year 4 was to provide language arts teachers and literacy coaches with a 1-day~ 
large-group training session during the school year and 12 in-school coaching visits. This was a 
decrease in professional development days from previous years.” The goals of these visits are to 
provide assistance to teachers through modeling and discussion of classroom practices such as 
developing vocabulary, using graphic organizers, establishing routines for silent reading, and 
improving reading comprehension strategies. The number of in-school visits remained the same in 
Year 4 as in Year 3 of the grant (5 visits in Year 1; 10 visits, Year 2; 12 visits, Year 3; 12 visits, Year 
4). 


NUA, the second professional development provider, provides professional development for 
teachers to support literacy across the content areas. Math, science, and social studies teachers were 
eligible to receive a 1-day” large-group training session during the school year and 15 in-school 
coaching visits. The amount of large-group professional development training decreased from Year 
3 to Year 4. However, the number of in-school visits (15 visits) has remained the same in all 4 years 


of the grant. 


For a description of and changes to the professional development model over the first 4 years of the 


grant please see Section 1.B.2.1 and Table 4. 


22 Rach day is a 5.5-hour session. 


3 For example in Year 3, a 4-day (4 hours each day) summer institute and two 1-day professional development sessions were held during the school 
year, and 12 in-school coaching visits per school during the school year. See Chapter 1 for a detailed table of changes to the professional 
development model. 


4 ach day is a 5.5 — hour session. 
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4.A.2 New Jersey City University Professional Development 


NJCU’s professional development was designed to introduce and reinforce the use of instructional 
strategies that enhance vocabulary development, fluency, and reading comprehension. The 
instructional strategies of NJCU’s large-group trainings primarily address the development of 
linguistic acquisition, establishing routines for silent reading, and improving reading comprehension 
and writing strategies. A binder of materials that included the NPS “Language Arts Literacy Policy 
and Practices for Elementary, Middle and Secondary Schools,” articles, strategies, graphic organizers, 
and sample activities on literacy strategies was distributed at each NJCU large-group professional 
development event. Daily feedback surveys also were used to ascertain the additional needs of 
participants; the workshop topics were revised based on the feedback to better address the identified 


areas of need. Sample workshop topics included the following: 


: How We Read —Understanding the complexity of the reading process via prior 
knowledge; grapho-phonemic, semantic, and syntactical strategies; linguistic 
competence; and vocabulary enrichment. 


: How We Assess and Teach Reading—Using assessment and diagnosis, miscue analysis, and 
strategies that promote success in reading, such as literature groups and circles; oral and 
silent reading best practices; purpose-setting; question-answer relationships; text 
annotation; note-taking; anticipation guides and post-reading reflection; double-entry 
journals; SQ3R; flowcharts, webs, and other graphic organizers; K-W-H-L-S; and 
personal dictionaries and vocabulary keepers. 


7 How We Structure Effective Literacy Programs—Understanding a brief history of literacy 
instruction, effective whole-class and small-group instruction, targeted instruction, 
phonemic awareness, guided reading, balanced literacy, and reading and writing across 
the curriculum. 


. Best Practices in Writing Instruction—Gaining a historical perspective on writing instruction, 
the writing process, spelling and vocabulary development, the reading—writing 
connection, time management and the writing process, and extending the writing 
process. 


: How to Promote Speaking and Listening Skills—Extending the reading—writing process to 
everyday conversation and enhancing the question/answer relationship. 


In Year 4, NJCU was contracted to provide a minimum of 12 on-site coaching visits to each school, 
focusing on the quality of literacy instruction through observation, demonstration, and coaching. 
These visits provide an important opportunity for teachers to observe modeling sessions based on 
site-specific instructional needs and participate in debriefing periods afterward. During the coaching 


visits, NJCU coaches observe language arts literacy teachers and provide modeling and assistance in 
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the literacy areas covered in the large-group trainings. The topics discussed and the practices 
modeled in the classroom include developing vocabulary, establishing routines for silent reading, 
identifying and using reading comprehension strategies, making reading—writing connections, 
responding to text with writing prompts, using 
graphic organizers, initiating summary writing, Thinking Maps Overview 
identifying major themes in texts, engaging in 
a J 2 eee Circle Map: Used for seeking context. This tool enables 

reading and writing of poetry, and developing students to generate relevant information about a topic 
as represented in the center of the circle. This map 


often is used for brainstorming, building both 
follows each lesson to allow coaches to describe vocabulary and comprehension. 


habits of revising and editing. A debriefing session 


what they see and identify important details that Bubble Map: Designed for the process of describing 
attributes. This map is used to identify character traits 
(language arts), cultural traits (social studies), properties 
NJCU coaches observe teachers as they implement (science), or attributes (mathematics). This map 
develops vocabulary and comprehension and, in doing 
so, builds fluency. 


foster advanced thinking. In subsequent visits, the 


the demonstrated lessons. 


Double Bubble Map: Used for comparing and 
contrasting, such as characters in a story, historical 


4 A.3 National Urban Alliance figures, or social systems. This map also is used for 
7 . prioritizing information within a comparison and 
Professional Development building comprehension. 


Tree Map: Enables students to do both inductive and 
deductive classification and is particularly useful in the 
sciences. Students learn to create general concepts, 
group workshops were to train teachers in cognitive main ideas, category headings, supporting ideas and 
details, merging literacy and content area skills to make 
meaning (comprehension). 


The purposes of the summer institute and large- 


strategies that focus on the teaching, learning, and 
assessment of advanced thinking; to break down Brace Mop seed iene Rina Re Tae Ne 
school isolation; to build effective school teams; physical relationships of an object. This map, like the 
Tree Map, is a visual imagery strategy endorsed by the 
Strategic Instruction Model (SIM) of the Center for 
meta-cognitive and affective component was to be Research on Learning, also noted in the Reading Next 
report as a strategy to develop comprehension. 


and to create a community of learners. A strong 


part of each workshop, encompassing such 
Flow Map: Used for showing sequences, order, 
timelines, cycles, actions, steps, and directions. This map 
bias; multiple intelligences; English language also develops comprehension and fluency skills, as 
relationships between events are clearly seen. 


instructional issues as ethnic, gender, and racial 


learners (ELLs); special needs students; and 
Multiflow Map: Used as a tool for seeking cause/effect 
relationships. The map expands when showing historical 
strategies intend to accelerate the cognitive skills causes and predicting future events and outcomes. This 
map increases comprehension. 


learning styles. NUA’s professional development 


that support literacy development through 


Bridge Map: Provides a visual pathway for creating and 
interpreting analogies. This map positively affects 
learning patterns of students, and address the comprehension, vocabulary, and fluency as analogical 
reasoning and metaphorical concepts for deeper 
content learning are developed. 


strategies that are brain based, reflect the cultural 


district’s standards and learning goals. 
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The primary tools NUA uses to connect the content area and literacy are Thinking Maps,” which 
NUA uses as a professional development foundation to assist students in constructing, creating, and 
communicating meaning in the content areas by developing vocabulary, comprehension, and 
associated fluency strategies. NUA professional development has tackled these identified skills by 
connecting them to theoretical research on how the brain develops and how students from urban 


backgrounds learn. 


NUA also promotes content literacy strategies that increase student achievement, as suggested by 
recognized adolescent literacy specialists (e.g., Kylene Beers, Janet Allen, Nancy Atwell, Tom 
Romano, Alfred Tatum, Michael Smith). These specialists agree that students must know the 
vocabulary of the content discipline, must access prior knowledge of the content or subject area, and 
must possess study skills such as note-taking in their predominant learning style to assist their ability 
to recall information from multiple sources. Students must bring skills in reading expository text 
rather than narrative text to the foreground in content disciplines, must monitor their understanding 
of the text and adjust speed and concentration to fit the difficulty of the text, and must possess 
techniques for organizing the information. In addition, they must have mastered basic skills of 
decoding, fluency, phonics, and comprehension—the learning to read skills—so they can now read 
to learn. The primary content literacy skills addressed in the NUA’s professional development are 
vocabulary, fluency, and comprehension developed through defining in context; describing; 
compating and contrasting; classifying; sequencing; cause and effect reasoning; part-whole 


relationships; and analogies. 


Teachers have been introduced to various literacy strategies during the first 4 years of the 
intervention. These strategies include taxonomies (literacy content), Circle Maps, Bubble and 
Double Bubble Maps, and Flow Maps (Thinking Maps). Additionally, teachers have been introduced 
to Brace Maps, Multiflow Maps, and Tree Maps while refining their use of the initial thinking maps. 
To reiterate NUA’s objective, the goal is to have students reach a point where they can proficiently 
explore and construct meaning from texts: “When students put language to work for them in 
content classrooms, it helps them to discover organize, retrieve, and elaborate what they are 
learning” (Vacca, 2000, p. 13). 


To reinforce the implementation of the instructional strategies covered in the large-group training, 
NUA mentors visited each Striving Readers school. Ten school-based sessions were to be conducted 
to demonstrate (and provide coaching relative to) the application of the strategies presented during 
the large-group workshops. In the demonstration lessons, NUA mentors focused on the three 


systems that exist in every classroom: the relationship of teacher to student, the relationship of 
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teacher to content, and the delivery system. Preceding each lesson, the mentor briefs the teacher on 


the lesson’s content, strategies, and rationale for selection of strategies. 


The on-site demonstration lessons were to be conducted with half of each school’s grades 6-8 
faculty in attendance during either the morning or afternoon sessions to minimize the need for 
substitute teachers. NUA’s demonstration lessons are designed to address the heterogeneous make- 
up of the classroom, to be conducted in front of faculty from the school, to use authentic 
instructional materials, to be cued to existing courses of study and curricular demands, and to vividly 
illustrate the significant differences in advanced-level thinking that the cognitive strategies would 
make possible. A debriefing session follows each lesson to allow observers to describe what they saw 
and identify important details that foster advanced thinking. After the demonstration lessons, the 
NUA design offers opportunities for teachers to practice what was observed. NUA mentors then 
use the peer coaching model to share with teachers what they observed and make additional 


comments. 


4.B Implementation Results, Year 4 


To determine the degree of fidelity to the whole-school intervention, multiple components were 
evaluated for each Striving Readers school. Subscores were developed to measure the extent to 


which each component was implemented. These components are the following: 


Large-group training 
NJCU 
NUA 
In-school coaching 
NJCU 


NUA 


Each of these components is discussed in the following sections. Year 4 results are presented, 
followed by a summary of implementation in Years 1—4 as well as a discussion of changes in 


implementation between Years 1 and 4. 
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4.B.1 Large-Group Implementation Results, Year 4 
4.B.1.1 NJCU Large-Group Training Participation, Year 4 


The first component of NJCU’s professional development for language arts literacy teachers was the 
large-group training session. In alignment with the long-term goals of the project (in particular the 
embodiment of literacy-focused pedagogy) language arts literacy teachers had the opportunity to 


attend one large-group training session held during the school year on January 27, 2010.” 


A total of 199 teachers were eligible to receive professional development from NJCU in Year 4. 
These eligible teachers have been categorized by their level of professional development activities. 


Table 70 defines the participation levels. 


Table 70. Participation categories for NJCU group training in Year 4 


Component Full participation No participation 


January Large-group 1 Day O days 


Table 71 provides the number and percentage of teachers at each of the levels of participation 
outlined in Table 70. 


Table 71. Number and percentage of NJCU-eligible teachers by level of participation in Year 4 


Number % 
Full participation 55 27.6 
No participation 144 72.4 
Total 199 100.0 


As Table 71 shows, 27.6 percent of eligible teachers attended the 1-day training and, 72.4 percent 
did not. Of these 144 teachers that did not attend the training, 111 were also eligible to attend the 
NUA training, provided on the same day. It is unknown whether these teachers were instructed to 
attend the NUA training, or another professional development training offered by the district (also 


on the same day). 


Some of the variation in teacher participation appears to reside at the school level. At the school 


level, the percentage of teachers receiving the full amount of NJCU professional development 


25 The training during the school year was 6 hours long. 
g s J 8: 
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ranges from 0 to 62.5 percent. A score was created to summarize the level of participation at the 
school level for the large-group trainings provided by NJCU. Depending on the percentage of 
teachers who had full participation in the large-group trainings, each school was given an overall 
score of 1-4. For example, if a school had 80 percent of its teachers reach full participation, it would 


26 


receive a high score of “4.” Table 72 shows how the score was calculated. 


Table 72. Criteria for participation in NJCU group training 


Percentage of teachers with full participation School participation score 
75-100% 4 (High) 
50-74% 3 (Moderate-to-High) 
25-49% 2 (Low-to-Moderate) 
0-24% 14 (Low) 


Table 73 presents the breakdown of participation by school. 


Table 73. Number and percentage of teachers in each school by participation category: NJCU, 


Year 4 
Total no. of % of teachers with % of teachers with School 
School teachers full participation no participation participation score 

School 1 8 62.5 37.5 Moderate-to-High 
School 2 4 50 50 Moderate-to-High 
School 5 11 18.2 81.8 Moderate-to-High 
School 9 9 55.6 44.4 Moderate-to-High 
School 3 5 40 60 Low-to-Moderate 
School 4 8 25 75 Low-to-Moderate 
School 6 13 46.2 53.8 Low-to-Moderate 
School 7 8 37.5 62.5 Low-to-Moderate 
School 8 8 37.5 62.5 Low-to-Moderate 
School 10 6 33.3 66.7 Low-to-Moderate 
School 12 7 28.6 71.4 Low-to-Moderate 
School 13 21 33.3 66.7 Low-to-Moderate 
School 14 9 33.3 66.7 Low-to-Moderate 
School 18 10 40 60 Low-to-Moderate 
School 11 24 12.5 87.5 Low 

School 15 14 14.3 85.7 Low 

School 16 5 Oo 100 Low 

School 17 16 12.5 87.5 Low 

School 19 13 O 100 Low 

Total/Average 199 30.5 69.5 Low-to-Moderate 


26 Full participation was calculated differently than in previous years because of the decrease in training days offered. 
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In Year 4, no school had all their eligible NJCU teachers attend the 1-day training session. Only 4 
schools (21 percent) achieved the second highest level of participating, and 10 schools (52.6 percent) 
achieved Low-to-Moderate participation. Five schools (26.3 percent) had the lowest level of 
participation, with 2 of these schools having none of their teachers attend the NJCU training 
session. Such low levels of participation can have serious implications for the likelihood of showing 
effects of the whole-school intervention. School-specific factors that might have caused such wide 
variation in attendance include staff transfers and communication about attendance for NJCU-led 
events. Additionally, with only 1 day of training offered during the school year, teachers may have 
had to choose from a variety of trainings or have been directed to attend a different training. With 
only one NJCU training offered in Year 4, teachers were placed in an “all or nothing” situation. 
Participation scores were either High or Low-to-Moderate with no Moderate-to-High option. In the 
past, teachers could receive a higher score if they had attended some, but not all, of the training; 


although in Year 4, there was no middle option. 


4.B.1.2 NUALarge-Group Training Participation, Year 4 


The first component of NUA’s professional development for content area teachers was the large- 


group training session on January 27, 2010.” 


A total of 278 teachers were eligible to receive professional development from NUA in Year 4. 
These eligible teachers have been categorized by their level of participation in the NUA professional 


development activities. Table 74 defines the participation levels. 


Table 74. Participation categories for NUA group training in Year 4 


Component Full participation No participation 


January Large-group 1 Day O days 


Table 75 provides the number and percentage of teachers at each of the levels of participation 
outlined in Table 74. 


27 The training during the school year was 6 hours. 
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Table 75. Number and percentage of NUA-eligible teachers by level of participation in Year 4 


Number % 
Full participation 98 35.3 
No participation 180 64.7 
Total 278 100.0 


As Table 75 shows, 35.3 percent of NUA-eligible teachers received the professional development 
training, while nearly 65 percent did not. It should be noted roughly 50 percent of teachers were 
eligible for both NUA and NJCU training. These teachers either taught both language arts and a 
content area subject (usually social studies), or they taught all subjects (usually special education or 
bilingual teachers). Of the 180 teachers who did not attend the NUA training, 93 were also eligible 
to attend the NJCU training held the same day. It is not known whether these teachers were 
instructed to attend the NJCU training or one of the other professional development sessions being 


offered by the district on the same day. 


Again, at least some of the variation in participation appears to reside at the school level. As shown 
in Table 77, across the 19 Striving Readers schools, the percentage of teachers receiving the amount 
of NUA professional development ranged from 0 to 62.5 percent. A score was created for the 
school level to summarize the extent of participation at the large-group trainings provided by NUA. 
Depending on the percentage of teachers who had full or moderate participation in the large-group 
trainings, each school was given an overall score of 1-4. For example, if a school had 80 percent of 
its teachers reach full and/or moderate participation, it would receive a high score of “4.” Table 76 


shows how the score was calculated. 


Table 76. Criteria for participation in NUA group training 


Percentage with full or moderate participation School participation score 
75-100% 4 (High) 
50-74% 3 (Moderate-to-High) 
25-49% 2 (Low-to-Moderate) 
0-24% 14 (Low) 


Table 77 provides the breakdown of participation by school. 
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Table 77. Number and percentage of teachers in each school by participation category: 


NUA, Year 4 
Total no. of % of teachers with % of teachers with School 
School teachers full participation no participation* participation score 
School 3 8 50 50 Moderate-to-High 
School 7 14 50 50 Moderate-to-High 
School 12 10 60 40 Moderate-to-High 
School 14 13 61.5 38.5 Moderate-to-High 
School 15 16 62.5 37.5 Moderate-to-High 
School 18 13 61.5 38.5 Moderate-to-High 
School 19 15 53.3 46.7 Moderate-to-High 
School 5 16 37.5 62.5 Low-to-Moderate 
School 6 21 38.1 61.9 Low-to-Moderate 
School 8 10 40 60 Low-to-Moderate 
School 10 10 30 70 Low-to-Moderate 
School 13 24 41.7 58.3 Low-to-Moderate 
School 16 8 37.5 62.5 Low-to-Moderate 
School 1 13 23.1 76.9 Low 
School 2 8 0 100 Low 
School 4 12 8.3 91.7 Low 
School 9 12 8.3 91.7 Low 
School 11 35 17.1 82.9 Low 
School 17 20 10 90 Low 
Total/Average 278 36.3 63.7 Low-to-Moderate 


* Low and no participation have been combined. 


In Year 4, no school achieved the highest level of participation, meaning no school had all the 
eligible NUA teachers attend the 1-day training. Seven schools (36.8 percent) achieved the second 
highest level of participating, earning a score of 3. Six schools (31.6 percent) achieved Low-to- 
Moderate participation, while another 6 (31.6 percent) had the lowest level of participation. Of these 
six schools, one school had no eligible teachers in the school attending the NUA training session in 
Year 4. Low levels of participation can have serious implications for the likelihood of showing 
effects of the whole-school intervention. School-specific factors that might have caused such wide 
variation in attendance include staff transfers and communication about attendance for NUA-led 
events. Additionally, with only 1 day of training offered during the school year, teachers may have 
had to choose from a variety of trainings or have been directed to attend a different training. With 
only one NUA training offered in Year 4, teachers were placed in an “all or nothing” situation. 


Participation scores were either High or Low-to-Moderate with no Moderate-to-High option. In the 


89 


past teachers could receive a higher score if they had attended some, but not all, of the training, 


although in Year 4 there was no middle option. 


4.B.2 


4.B.2.1 


In-School Coaching Implementation Results, Year 4 


NJCU Participation 


The second component of NJCU’s professional development for language arts teachers was in- 


school coaching visits. The plan was for NJCU coaches to visit all 19 Striving Readers schools, 
starting in September 2009 and ending in May 2010. Each of the 19 schools was visited by a NJCU 


coach 12 times.” 


A score was calculated for each school based on the number of coaching visits received during Year 


4. Table 78 provides the scoring criteria, while Table 79 presents the coaching score. 


Table 78. Criteria for in-school coaching visits for NJCU 

% of intended visits School score 
75-100% 4 (High) 
50-74% 3 (Moderate-to-High) 
25-49% 2 (Low-to-Moderate) 
0-24% 14 (Low) 


°8 As contracted through the Striving Readers grant, NJCU was expected to make 5 visits to Striving Readers schools in Year 4; however, Title I funds 
were used to subsidize the number of visits for a total 12 visits per school. 
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Table 79. Number of coaching visits received by school and resulting coaching score: NJCU, 


Year 4 
School Number of coaching visits School coaching score 
School 1 12 High 
School 2 12 High 
School 3 12 High 
School 4 12 High 
School 5 12 High 
School 6 12 High 
School 7 12 High 
School 8 12 High 
School 9 12 High 
School 10 12 High 
School 11 12 High 
School 12 12 High 
School 13 12 High 
School 14 12 High 
School 15 12 High 
School 16 12 High 
School 17 12 High 
School 18 12 High 
School 19 12 High 
Average 12 High 


As Table 79 shows, all of the schools (100 percent) in Year 4 received all of the intended coaching 


visits laid out by the intervention model. 


It is possible that the coaching visits were able to mitigate the low participation in the group 
sessions. More than 35.7 percent of teachers received at least 1 hour of in-school coaching visits in 
Year 4. On average, teachers got an extra 6.6 hours of instruction from NJCUs’ coaching visits (see 


Table 80). 


There was large variation in the amount of coaching houts at each school (1.87—18.10 hours). This 
range can be attributed to multiple factors such as different numbers of teachers at each school and 
the level of need of each individual teacher. Coaching was provided on an as-needed basis, so no set 
number of hours was required for each teacher. Some teachers may have needed more in-school 


coaching hours than others to feel comfortable with the techniques. Therefore, the average hours of 


91 


coaching received is provided as description but does not necessarily reflect that certain schools had 


higher or better levels of participation (see Table 80). 


Table 80. Average number of NJCU coaching hours received by school in Year 4 


No. of Receipt of coaching visits: Number of teachers 

Total no. of coaching Avg. hours with following participation levels* 
School teachers visits rec’d 0 hrs 1-15hrs 16-30hrs 31+hrs 
School 1 8 12 7.77 3 5 Oo Oo 
School 2 4 12 11.75 1 2 1 0 
School 3 5 12 9.01 3 1 O 1 
School 4 8 12 4.63 6 Oo 2 Oo 
School 5 11 12 3.32 8 2 4 0 
School 6 8 12 7.63 3 3 2 O 
School 7 13 12 1.87 10 3 Oo Oo 
School 8 8 12 6.39 1 1 1 
School 9 9 12 18.10 4 1 1 3 
School 10 6 12 6.89 4 Oo 2 Oo 
School 11 24 12 2.53 18 5 1 0 
School 12 7 12 7.33 5 1 0 1 
School 13 21 12 5.38 13 4 4 Oo 
School 14 9 12 4.90 O 2 O 
School 15 14 12 5.09 6 1 0 
School 16 5 12 9.40 0 0 1 
School 17 16 12 3.62 11 3 2 O 
School 18 10 12 4.75 7 3 O 0 
School 19 13 12 4.16 9 2 2 Oo 
Total/Average 199 12 6.55 128 42 22 


* It is impossible to determine the expected number of hours per school visit, because the visits were tailored to the specific needs of 
each school and the type of training provided (such as group sessions or individual demonstration lessons) and sometimes depended 
on the level of substitute coverage obtained. In addition, NPS did not require or specify the number of visits needed during the school 
year. 


4.B.2.2 NUA Participation 


The second component of the whole-school intervention is the in-school visits provided by the 
NUA mentors. The plan was for NUA to visit all 19 Striving Readers schools in the 4th year of the 
grant, starting in September 2009 and ending in June 2010. Each school was to be visited by a NUA 
mentor for 15 days. Each school received an average of 14.1 visits. The number per school ranged 


from 13 to 15 visits. A score was calculated for each school based on the number of coaching visits 
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received during Year 4. Table 81 provides the scoring criteria, while Table 82 presents the coaching 


score. 


Table 81. Criteria for in-school coaching visits for NUA 


% of intended visits School score 
75-100% 4 (High) 
50-74% 3 (Moderate-to-High) 
25-49% 2 (Low-to-Moderate) 
0-24% 1 (Low) 


Table 82. Number of coaching visits received by school and resulting coaching score: NUA, Year 4 


School Number of coaching visits School coaching score 
School 1 14 High 
School 2 14 High 
School 3 14 High 
School 4 14 High 
School 5 14 High 
School 6 13 High 
School 7 15 High 
School 8 14 High 
School 9 15 High 
School 10 14 High 
School 11 15 High 
School 12 13 High 
School 13 14 High 
School 14 14 High 
School 15 14 High 
School 16 14 High 
School 17 15 High 
School 18 14 High 
School 19 15 High 
Average 14,12 High 
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All 19 schools received all or nearly all of the coaching visits intended by the intervention. It is 
possible that the coaching visits were able to mitigate the low participation in the group sessions. 
Nearly 70 percent of NUA teachers received at least 1 hour of in-school coaching in Year 4 and 16.1 
percent of teachers received 16 hours or more of in-school coaching. Table 83 shows that, on 
average, teachers got an extra 9.4 hours of instruction from NUA coaching visits. Similar to NJCU, 
the number of hours individual teachers received was based on individual need. Teachers who 
received fewer hours of in-school coaching may not have required as intensive coaching as others, 


thus causing the wide variation in individual hours of coaching received. 


Table 83. Average number of NUA coaching hours received by school in Year 4 


No. of Receipt of coaching visits: Number of teachers 
Total no. of coaching Avg. hours with following participation levels* 
School teachers visits rec'd 0 hrs 1-15hrs 16-30hrs 314+hrs 

School 1 13 14 4.76 5 7 1 

School 2 8 14 9.75 1 5 2 O 
School 3 8 14 11.75 2 2 4 0 
School 4 12 14 6.80 4 7 1 0 
School 5 16 14 13.42 4 5 6 1 
School 6 14 13 7.19 5 12 2 0 
School 7 21 15 9.62 0 15 1 0 
School 8 10 14 14.25 3 5 O 
School 9 12 15 8.92 9 0 2 
School 10 10 14 15.08 2 5 0 
School 11 35 15 2.12 21 14 O 0 
School 12 10 13 22.34 1 3 2 4 
School 13 24 14 3.83 7 16 1 0 
School 14 13 14 9.44 4 6 3 O 
School 15 16 14 11.53 2 13 1 0 
School 16 8 14 8.17 2 5 1 0 
School 17 20 15 2.50 7 13 O 0 
School 18 13 14 12.03 1 11 1 0 
School 19 15 15 5.42 5 8 2 0 
Total/Average 278 14.12 9.42 85 148 38 7 


* It is impossible to determine the expected number of hours per school visit, because the visits were tailored to the specific needs of 
each school and the type of training provided (such as group sessions or individual demonstration lessons) and sometimes depended 
on the level of substitute coverage obtained. In addition, NPS did not require or specify the number of visits needed during the school 
year. 
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4.B.2.3 RTC In-School Coaching Participation 


In addition to the in-school support from developers, language arts teachers also received in-school 
coaching support from the Striving Readers RTCs. This support was provided on an as-needed 
basis. Between September 2009, and June 2010, the RTCs visited all 19 schools. Each Striving 
Readers school was visited by an RTC an average of 18 days, ranging from 4 to 28 visits (see Table 
84). 


Table 84. Number of RTC coaching visits received by school in Year 4 


School Number of RTC coaching visits 
School 1 23 
School 2 16 
School 3 31 
School 4 31 
School 5 16 
School 6 16 
School 7 12 
School 8 17 
School 9 15 
School 10 27 
School 11 14 
School 12 17 
School 13 20 
School 14 20 
School 15 14 
School 16 18 
School 17 28 
School 18 4 
School 19 14 
Average 18.6 


During these visits, RTCs worked with teachers on various whole-school activities, such as offering 
classroom support, coaching, modeling, offering assistance with student work, and using student 


data to inform instruction.” In addition, they assisted in preparing for the New Jersey Assessment of 


20RTCs ate required to complete an “RTC Log” each time they visit a school. However, in Year 1, the log did not allow the research team to 
differentiate between visits to support the whole-school intervention and visits to support the targeted intervention. This was corrected in Year 2 and 
was used in Year 3 and 4. The form is provided in Appendix E. 
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Skills and Knowledge (NJASWS), the General Education Provisions Act (GEPA) initiative, and 


standards-based lessons. Often RTCs worked on multiple activities during one visit. 


4.B.3 Whole School Intervention - Participation Summary, Year 4 


A summary scale for Year 4 was developed to describe the professional development components 
across the four years of the project. An overall implementation score and level of implementation 
were calculated for each school in the study. Table 85 lists the definitions for the school-level 
implementation, which are based on the implementation scores for group sessions and coaching 
visits for NJCU. Table 86 provides each school’s score for the multiple facets of the professional 
development for the whole-school intervention—the group training sessions and the in-school 


coaching visits—in the NUA and the NJCU intervention models. 


Table 85. Criteria for teacher participation in whole-school professional development activities 


Average implementation score Overall implementation level 
4.0 High 
3.0-3.9 Moderate-to-High 
2.0-2.9 Moderate 
0-1.9 Low 


As Table 86 shows, no school achieved full implementation of all four components. However, 15 
schools had Moderate-to-High levels of implementation for the whole-school intervention. The 
remaining four schools all had moderate levels of implementation, taking into account all 


components of the whole-school professional development. 
It should be noted that the relatively high average levels of participation are related more to the high 
levels of whole-school coaching than to high levels of teacher participation in the group training 


session. Even where teacher participation in the group training was poor, the developers (NUA and 


NJCU) compensated through multiple in-school visits. 


4.C Barriers to Whole-School Implementation, Year 4 


The most significant difference between the whole-school intervention “as planned” and “as 


implemented” was the low level of participation of teachers in both NJCU and NUA large-group 
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trainings. The many teachers who received very no large-group training posed a serious problem for 
implementation of the whole-school intervention. Whole-school intervention effects may not have 
been found because of low participation rates of teachers rather than the ineffectiveness of the 


intervention itself. 


Table 86. School-level summary scores for participation in whole-school professional 
development activities in Year 4 


Implementation scores by component 


NUA NJCU Summary 
Large-group In-school Large-group In-school Average implementation 
School training coaching training coaching score scores 

School 1 1 4 3 4 3 Moderate-to-High 
School 2 1 4 3 4 3 Moderate-to-High 
School 3 3 4 2 4 3.25 Moderate-to-High 
School 5 2 4 3 4 3.25 Moderate-to-High 
School 6 2 4 2 4 3 Moderate-to-High 
School 7 3 4 2 4 3.25 Moderate-to-High 
School 8 2 4 2 4 3 Moderate-to-High 
School 9 1 4 3 4 3 Moderate-to-High 
School 10 2 4 2 4 3 Moderate-to-High 
School 12 3 4 2 4 3.25 Moderate-to-High 
School 13 2 4 2 4 3 Moderate-to-High 
School 14 3 4 2 4 3.25 Moderate-to-High 
School 15 3 4 1 4 3 Moderate-to-High 
School 18 3 4 2 4 3.25 Moderate-to-High 
School 19 3 4 1 4 3 Moderate-to-High 
School 4 1 4 2 4 2.75 Moderate 
School 11 1 4 1 4 2.5 Moderate 
School 16 2 4 1 4 2.75 Moderate 
School 17 1 4 1 4 2.5 Moderate 
Average 2.05 4.00 1.95 4.00 3 Moderate-to-High 


Based from feedback from district administrators, several additional barriers to participation in Year 
4 emerged. District administrators noted that with increased teaching days added to the school 
calendar, there was a reduction in professional development days allotted in the district. With fewer 
professional development days, teachers did not have the same opportunity as in previous years to 


attend trainings. 
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There were also scheduling conflicts which posed obstacles for teachers. NPS sets aside days for 
district-wide professional development during the school year, and all training (not just for the 
Striving Readers grant) occurs on these days. Therefore, NPS must decide whether the teacher 
should attend the Striving Readers professional development sessions, the other sessions also 
scheduled, or remain in the school for departmental meetings. Moreover, a number of teachers are 
eligible to attend both the NUA and the NJCU sessions. Therefore, these dual-eligible teachers must 
choose one curriculum partner over the other. In previous years, these dual-eligible teachers may 
have been able to attend 1 day of NUA training and on another professional development day 
attend a NJCU training; however, with only one training day held (and NUA and NJCU held on the 


same days) teachers had to make a choice between them. 


It was hoped that in Year 4 these trainings could be held on non-conflicting days. Unfortunately, it 
was not logistically possible. NPS has only a certain number of professional development days 


allotted, and multiple initiatives take place throughout the district that require large-group trainings. 


In addition to low participation by the teachers in the professional development sessions, teacher 
turnover from Year 3 to Year 4 also was high (see Table 87). Despite direction from NPS asking 
principals not to reassign Striving Readers teachers, teacher turnover was 23 percent. Table 87 


describes teacher turnover from Year 3 to Year 4. 


Table 87. Teacher turnover from Year 3 to Year 4 


Year 3 teacher Year 4 teacher Number of teachers % 
Yes Yes 260 77.2 
Yes No 77 22.8 
No Yes 77 - 


Table 88 describes how many teachers have taught in Striving Readers classrooms for 1 or more 
years of the grant. Almost a quarter of teachers (149 teachers) have remained in the project over the 
first 4 years, while 36.6 percent of teachers (220 teachers) have only remained in Striving Readers 
classrooms for 1 out of the 4 years. However, it is encouraging that over 40 percent of teachers have 
remained for at least 3 of the 4 years of the grant. Teachers have shown a range of movement in and 
out of Striving Readers classrooms, with some beginning the grant in Year 1, leaving, and 
subsequently returning to a Striving Readers classroom at various time points. This movement of 
teachers in and out over the 4 years of the grant may contribute to a lack of findings for the whole- 


school intervention. 
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Table 88. Number of teachers by years taught in Striving Readers Classroom 


How many years taught in a % of total Striving Readers 
Striving Readers Classroom Number of teachers Teachers from all 4 Years 
All 4 Years 149 24.8 
3 Years 94 15.6 
2 Years 138 23.0 
1 Year 220 36.6 
Total 601 


The issue of principal turnover, was a particular challenge in Year 2, but was somewhat resolved in 
Years 3 and 4. Principal turnover from Year 2 to Year 3 was 15.8 percent, while turnover from Year 
3 to Year 4 was 21.1 percent with four new principals taking over in Year 4 (see Table 89). However, 
only six principals have remained consistent from Year 1 through Year 4. Still, an additional seven 
principals have been at their respective school from Year 2 through Year 4 of the grant, offering 
some consistency in 13 of the 19 Striving Readers schools in the 3 most recent years of the grant. 
Principal turnover could have consequences for implementation. For example, if new principals are 
not familiar with or buy into the whole-school intervention, support of teacher participation in 


trainings could be hindered. 


Table 89. Principal turnover from Year 3 to Year 4 


Year 3 principal Year 4 principal Number of principals % 
Yes Yes 15 78.9 
Yes No 4 21.1 


Another challenge in Year 4 was a reduction in RTCs available to support Striving Readers. Two of 
the five RTCs remained the same from the previous years; however, there were changes in the 
remaining three RTCs. One of the RTCs was on maternity leave during Year 4 and another was on 
extended medical leave. The third RTC was new to the project. This decrease in RTCs may have 
been the reason for the decrease in RTC visits in the school from Year 3 (average of 28 visits per 


school) to Year 4 (average of 18 visits per school). 


Some serious challenges remain in regard to implementation of the whole-school intervention. 
Despite these challenges, no changes were made between Year 3 and Year 4 to increase participation 
in large-group training. Conflicting initiatives in the District and a decrease in teacher professional 


development days, in addition to teacher and principal turnover made increasing participation 
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difficult. Sign in sheets and the In-School Professional Development Form (Appendix D) continued 


to be implemented in Year 4. These assisted in recording teachers’ participation in training. 


4.D Years 1-4 Implementation Summary 
4.D.1 | Whole-School Training Participation, Years 1-4 Summary 


The whole-school intervention consisted of both large-group professional development and in- 
school coaching visits. The degree of implementation in Years 1 through 4 was determined by 
examining the extent of teacher participation in the large-group training activities provided by NUA 
and NJCU. Schools were given a participation score based on the percentage of eligible teachers 
who attended the relevant large-group training sessions. The scores ranged from 1 to 4. For 
example, in the NUA column, a school was given a score of “4” if more than three-quarters of all 
eligible content area teachers attended the NUA large-group professional development sessions. 
Similarly, a score of “1” was assigned to a school where less than a quarter of teachers attended. An 
average score then was computed for each school based on the attendance at NUA and NJCU large- 
group sessions. Based on the average of participation scores, each school was assigned a 
participation rating level: Low, Low-to-Moderate, Moderate-to-High, or High (see Table 90). Table 
91 provides a comparison of large-group participation scores in Years 1 through 4. It is important to 
note that in Year 4, there were considerably fewer training opportunities for teachers. No summer 
institutes were held and only 1 day of large-group training was held during the school year. In 
previous years, teachers had the opportunity to attend a 3-4 day summer institute and 2—3 days of 


large-group trainings during the school year. 


Table 90. Criteria for participation in large-group professional development activities 


Average score School participation rating 
3.1-4 High 
2.1-3 Moderate-to-High 
1.1-2 Low-to-Moderate 
0-1 Low 
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Table 91. Year 1, 2, 3, and 4 ratings by school on teacher participation in large-group professional development activities 


Year 1 Year 2 Year 3 Year 4 Change Change 
Avg. Avg. Avg. Avg. from from 
NUA NJCU Score NUA NJCU Score NUA NJCU Score NUA NJCU Score Year3to Yearito 


LOL 


School (4-4) (4-4) (44) (4-4) (44) (44) (44) (44) (44) (4-4) (4-4) (4-4) Year4 Year4 
School1 3 1 1.5 4 2 3 3 2 25 1 3 2 0.5 0.5 
School2 = 1 1 1 3 1 2 2 3 25 1 3 2 0.5 1 
School3 3 3 3 1 1 1 2 2 2 3 2 2.5 0.5 0.5 
School4 2 2 2 3 1 2 2 2 2 1 2 1.5 0.5 0.5 
School5 3 1 1.5 3 1 2 4 4 4 2 3 2.5 1.5 1 
School6 2 3 2.5 2 1 1.5 3 3 3 2 2 2 4 0.5 
School7 2 2 2 2 1 1.5 1 2 15 3 2 2.5 1 0.5 
School8 2 3 2.5 3 1 2 3 2 25 2 2 2 0.5 0.5 
School9 = 1 3 1.5 2 2 2 2 3 25 1 3 2 0.5 0.5 
School10 2 3 2.5 2 1 1.5 2 1 15 2 2 2 0.5 0.5 
School11 2 3 2.5 2 1 1.5 2 3 25 1 1 1 1.5 1.5 
School12 3 3 3 2 2 2 2 2 2 3 2 2.5 0.5 0.5 
School13. 1 2 1.5 1 1 1 3 1 2 2 2 2 0 0.5 
School14 2 2 2 3 2 2.5 3 2 25 3 2 2.5 0 0.5 
School15 2 2 2 1 2 1.5 3 3 3 3 1 2 “4 0 
Schooli6 3 2 2.5 4 1 2.5 3 1 2 2 1 1.5 0.5 “4 
School17 2 1 1.5 1 1 1 2 3 25 1 1 1 1.5 0.5 
School18 3 3 3 3 1 2 1 1 1 3 2 2.5 1.5 0.5 
School19 1 1 1 1 1 1 1 1 1 3 1 2 1 1 
Average 2.41 216 205 226 126 #1476 232 2416 224 205 195 200 024 0.05 


Over the 4 years of the project, overall large-group training scores ranged from the Low-to- 
Moderate category (Years 2 and 4) to Moderate-to-High (Years 1 and 3). Overall scores decreased 
from Year 3 to Year 4 by .24 points, however, the overall decrease from Year 1 to Year 4 was 
relatively small (.05 points) as can be seen in Figure 6. As discussed previously in section 4.C., there 
have been many barriers that have kept teachers from participating in the large-group trainings. It 
appeared in Year 3 that participation was at its best, but in Year 4 with the reduction in teacher 
training days and other initiatives offering training in the district on the same day, this may be a 


reason for the decrease from the High in Year 3 to Year 4. 


Figure 6. Average school-level ratings for whole-group participation, Years 1-4 


3.5 


2.5 


Year 1 Year 2 Year 3 Year4 


Table 92 shows the number and percentage of schools who achieved each of the four participation 
categories over the 4 years of the project. Year 3 had the highest number of schools in the 
Moderate-to-High category and one school even achieved the High rating. However in Year 4 the 
percentage of schools who achieved Moderate-to-High ratings decreased. The number of schools 


receiving a Low rating remained small (two schools) in Year 4. 
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Table 92. Summary of Years 1 through 4 school participation in large-group professional 
development activities by average rating 


Year 1 Year 2 Year 3 Year 4 


Number Number Number Number 
of Percentage of Percentage of Percentage of Percentage 
schools ofschools schools ofschools schools ofschools’ schools’ of schools 


Low 2 11 4 21 2 11 2 14 
Low to 9 47 12 63 7 37 11 58 
moderate 

Moderate- 8 42 3 15 9 47 6 32 
to-High 

High 0 Oo Oo Oo 1 5 Oo 0 


Overall, NUA large-group participation scores increased over the first 3 years of the grant but 
decreased in Year 4 (see Figure 7). In Year 1, the average score was 2.11, but it increased to 2.26 in 
Year 2 and 2.32 in Year 3. In Year 4 the score decreased to 2.05 but this did not lead to a change in 
the participation category (all four scores are in the Moderate-to-High range). A similar trend was 
not found for NJCU participation in large-group training as training participation varied over the 
four years of the grant. In Years 1 and 3, participation ratings were in the Moderate-to-High 
category; in Years 2 and 4, however, participation was in the Low-to-Moderate category. Again, the 
reduction in available teacher training days in Year 4 and other District wide initiatives offering 


training on the same day may have led to the decrease in participation in the large-group trainings. 


Figure 7. Changes in participation among Years 1, 2, 3, and 4 
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4.D.2 In-School Coaching Participation, Years 1-4 Summary 


The level of teacher support provided by the curriculum developers is calculated by examining the 
number of in-school visits made by NUA and NJCU. Based on the average of in-school 
participation scores, each school was assigned a participation rating level: Low, Low-to-Moderate, 
Moderate-to-High, or High (see Table 93). Table 94 provides each school with a score for these in- 
school visits. Each school’s score is based on the number of visits received compared to the number 
that was anticipated. For example, in the NUA column, a school is given a score of “4” if it recetved 
at least three-quarters of the designated coaching visits. An average score then is computed for each 
school based on the NUA and NJCU components. 


Table 93. Criteria for in-school coaching participation 


Average score School participation rating 
3.1-4 High 
2.1-3 Moderate-to-High 
1.1-2 Low-to-Moderate 
0-1 Low 
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Table 94. Years 1 through 4 ratings by school on receipt of in-school teacher support 


SOL 


Year 1 Year 2 Year 3 Year 4 Change Change 
from from 
Avg. Avg. Avg. Avg. Year3to Yearito 
School NUA NJCU Score NUA NJCU Score NUA NJCU Score NUA NJCU Score Year 4 Year 4 

School 1 4 4 4.0 4 4 4 4 4 4 4 4 4 0 0 
School 2 4 4 4.0 4 4 4 4 4 4 4 4 4 0 0 
School 3 4 4 4.0 4 4 4 4 4 4 4 4 4 0 ce) 
School 4 4 4 4.0 4 4 4 4 4 4 4 4 4 (¢) (0) 
School 5 4 4 4.0 4 4 4 4 4 4 4 4 4 0 0 
School 6 4 4 4.0 4 4 4 4 4 4 4 4 4 0 0 
School 7 4 4 4.0 4 4 4 4 4 4 4 4 4 0 0 
School 8 4 2 3.0 4 4 4 4 4 4 4 4 4 (0) 1 
School 9 4 4 4.0 4 4 4 4 3 3.5 4 4 4 0.5 0 
School 10 4 3 3.5 4 4 4 4 4 4 4 4 4 0 0.5 
School 11 4 4 4.0 4 4 4 4 3 3.5 4 4 4 0.5 0 
School 12 4 2 3.0 4 4 4 4 4 4 4 4 4 fe) 1 
School 13 4 1 2.5 4 4 4 4 4 4 4 4 4 0 1.5 
School 14 4 1 2.5 4 4 4 4 4 4 4 4 4 0) 1.5 
School 15 4 4 4.0 4 4 4 4 4 4 4 4 4 0 0 
School 16 4 3 3.5 4 4 4 4 4 4 4 4 4 0 0.5 
School 17 4 4 4.0 4 4 4 4 4 4 4 4 4 0 0 
School 18 4 1 2.5 4 1 2.5 4 4 4 4 4 4 ce) 1.5 
School 19 4 1 2.5 4 2 3 4 4 4 4 4 4 (0) 1.5 
Average 4.00 3.05 3.53 4.00 3.74 3.87 4.00 3.89 3.95 4.00 4.00 4.00 0.05 0.47 


Over the 4 years of the project, the overall in-school coaching score increased. In Year 1 the score 
overall score was 3.53 and by Year 4 the average score was 4.0, the highest score available. From 
Year 3 to Year 4, a slight increase of .05 occurred in the in-school coaching scores. Seventeen 
schools remained the same (again mainly because participation was already as high as possible) while 
two other schools showed increases in in-school coaching receipt. No schools decreased in in- 


school coaching scores from Year 3 to Year 4. 


In changes between Year 1 and Year 4, in-school coaching receipts increased overall by .47. Eleven 
schools remained the same (already at the highest level of implementation). Eight other schools 


increased their scores between Years 1 and 4. 


In-school coaching scores have been consistently high in all 4 years. In particular, NUA scores have 
been at the highest level since the start. NJCU scores were in the High range during the first year, 
but have steadily improved over the 4 years. In Year 4 NJCU, the average NJCU score was a 4, the 
highest score attainable. Figure 8 shows the increase in scores for NUA and NJCU over the 4 years 
of the project. 


Figure 8. | Changes in in-school support among Years 1, 2, and 3 
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Effect of Whole-School Intervention 


5.A Introduction 


The goal of Newark Public Schools (NPS) whole-school intervention is to improve students’ ability 
to “read to learn” across multiple content areas. The whole-school intervention is designed to train 
teachers to better integrate different learning strategies within the district’s core literacy program for 
middle-grade students. To this end, the intervention provides professional development to bolster 
the literacy knowledge of teachers of grades 6, 7, and 8 in whole-group settings and to provide direct 
coaching support using in-school visits. These professional development and support activities are 
conducted by experts from New Jersey City University (NJCU) and the National Urban Alliance 
(NUA). In this chapter, we describe the evaluation conducted to examine the impact of whole- 


school intervention on students’ literacy achievement. 


5.B Research Design 
5.B.1 Treatment and Comparison Group 


Because teachers in all 19 Striving Readers schools recetved the whole-school intervention, a 
randomized experimental design for this evaluation was not possible. Therefore, a quasi- 
experimental research design was adopted, with 19 other middle schools in the district serving as a 
compatison group. State test scores from students in grades 6—8 with were aggregated to the school 
level for the analyses. Treatment and comparison schools then were compared using a modified 
short interrupted time series model. This relatively new analytic approach to study whole-school 


reforms evaluates a program model when only a few years of data are available (Bloom, 2003). 


5.B.2 Measures 


The key measure in the whole-school intervention is the state’s standardized exam, the New Jersey 
Assessment of Skills and Knowledge (NJASK). The Language Arts Literacy (LAL) section of the 
exam is composed of both literacy/reading and writing sections. The analyses of student NJASK 


107 


scores focus on the connection between the implementation of the professional development 
intervention and student literacy achievement. To evaluate this relationship, the results of the 
NJASK LAL test were used as the measure of student learning. In the 2005—06, 2006-07, 2007-08 
and 2008-09 school years, students took the NJASK near the end of the year, typically in late April 
and early May. The short interrupted time series model is premised on comparing the performance 
of schools before and after the intervention was put in place. In this case, the NJASK scores from 
2005—06 provide the preprogram performance data. The next 3 years then provide the post-program 
implementation performance data, which allows a comparison to see if any changes took place after 


the whole-school intervention was started. 


In addition to achievement scores, school demographic data at baseline were also downloaded from 
the New Jersey State Department of Education (NJSDE) website. Data were collected on basic 
school level demographic characteristics: gender (percent male), ethnicity/race (percent African 
American and percent Hispanic), and special education (percent of students with an Individualized 


Education Program [IEP]). 


5.B.3 Analysis Model Specifications 


The fact that all teachers in the 19 Striving Readers schools received the whole-school treatment 
presented substantial challenges for this study. When the treatment and comparison groups are not 
randomly assigned for the intervention, serious threats can be introduced to the internal validity of 
the evaluation design. In addition, the NJASK test was changed during the 2006-07 school year, 
making LAL scores incomparable over time. A lack of student-level data for the comparison schools 


also reduced the number of available modeling alternatives. 


To deal with these issues, a mixed model regression approach, based on the interrupted time series 
(ITS) concept, was designed for the NJASK evaluation. The primary goal of this model is to 
leverage the availability of multiple years of data to factor out some of the validity issues raised by 
the lack of randomization. The model does this by comparing grade cohorts of treatment and 
compatison groups over time focusing specifically on the year-to-year gains in scale score averages. 
It is essentially a difference-in-difference model, because it focuses on the difference between 
treatment and comparison groups at different points in time. All of the middle schools in the district 
are included as a comparison group for two reasons. First, by including a larger number of schools 
in the analysis, the power of the models to detect significant differences is increased. Second, we 


assume that the homogeneity of the district makes all of the middle schools relatively similar, such 
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that any external factors that affected test scores, other than the whole-school intervention, can be 


effectively controlled for by the difference-in-difference model approach. 


The mixed model approach also allows the inclusion of covariates for observable differences in the 


demographic composition of schools as well as a random intercept to vary across the groups. 


Perhaps most importantly, it also takes into account the alteration of the NJASK exam by looking 


separately at the score changes from baseline to Year 1, from Year 1 to Year 2 and from Year 2 to 


Year 3. Whatever effect the changes to the test had on test scores is assumed to have affected all 


schools equally. The equation is specified as follows: 


Y, = B+ B,Urt,)+ (irl, )+ B,(7t, *Yrl,)+ 8,(7r2,,)+ Bt, *¥r2,,)+ B73) + 


M 
B, (Trt ,*¥r3,,)+ > Boom (Zu) + Moy + &% 


m=1 


is the school-man LAL scote at time point 7 (=0,1,2,3) of the /” school (=1,...45) 

= 1 if treatment school, = 0 if comparison 

= 1 if measurement from Year 1, = 0 else 

= 1 if measurement from Year 2, = 0 else 

= 1 if measurement from Year 3, = 0 else 

is the m” (w=1,...M) school-level covariate measured at baseline (including whether a school 
received the READ 180 targeted treatment) 

is a random intercept term for the /” school assumed distributed N(0, 7’) 

is the residual error of the * measurement from the conditional mean of the 7” school 
assumed distributed N(0, 1°) 

is the estimate of the mean difference between treatment and comparison schools at baseline 
is the estimate of the mean difference between baseline and Year 1 for comparison schools 

is the estimate of the mean difference between treatment and comparison schools in Baseline 
to Year 1 growth (i.e., treatment effect of 1 year of treatment) 

is the estimate of the mean difference between baseline and Year 2 for comparison schools 

is the estimate of the mean difference between treatment and comparison schools in Baseline 
to Year 2 growth (i.e., treatment effect of 2 years of treatment) 

is the estimate of the mean difference between baseline and Year 3 for comparison schools 

is the estimate of the mean difference between treatment and comparison schools in Baseline 
to Year 3 growth (i.e., treatment effect of 3 years of treatment) 


A statistically significant result for 8; would indicate that the whole-school intervention treatment 


schools had greater gains on the NJASK LAL test between baseline and Year 1 than the comparison 


group schools, while a statistically significant result for 8; and 8, would indicate that the whole- 


school intervention treatment schools had greater gains on the NJASK LAL test between baseline 


and Year 2, and between baseline and year 3 than the comparison group schools, respectively. 
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5.C Data Analysis and Results 
5.C.1 Baseline Equivalence Between the Treatment and Control Groups 


Baseline descriptive statistics have been analyzed to determine whether significant difference existed 
between the treatment and the control groups before the whole-school intervention. Tables 95 and 
96 show that although there are some differences between treatment and comparison schools at 
baseline, none of these differences are statistically significant. Therefore, based on our T-test results, 


the whole-school treatment group and the comparison group were equivalent at baseline. 


Table 95. Baseline equivalence - NJASK LAL test scores 


Treatment Schools Comparison Schools 
Average Number of Average Number of T Value 
Score schools Score schools (Pr > [t]) 
NJASK LAL 202.3 16* 207.2 13** 1.76 (0.10) 


* School-level NJASK LAL data were missing for 3 of the schools in the treatment group 


** School-level NJASK LAL data were missing for 6 of the schools in the control group 


Table 96. Baseline equivalence - demographic variables 


T Val 
Demographic variables Treatment schools Comparison schools (Pr Z ap 
Black N=19 65% N=19 60% -0.45 (0.65) 
Hispanic N=19 32% N=19 25.3% -0.72 (0.65) 
Male N=19 45.2% N=19 47.9% 1.06 (0.290 
Special Ed N=19 10.7% N=19 8% -1.2 (0.23) 


5.C.2 Whole-School Treatment Effect on Students 


Table 97 provides descriptive statistics for the NJASK LAL analyses of the whole-school impact 
evaluation. As noted previously, the analyses were conducted at the school level, meaning that the 


statistics presented are school averages for each variable. 
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Table 97. NJAK LAL Scores for the Treatment and Comparison Schools (Years 1-3) 3° 


Treatment Schools Comparison Schools 
Average Number of Average Number of T Value 
Year Score schools Score schools (Pr > |t]) 
2005-2006 (Baseline) 202.3 16 207.2 13 1.76 (0.10) 
2006-2007 (Year 1) 187.3 19 192.9 19 1.28 (0.21) 
2007-2008 (Year 2) 187.2 19 192.9 16 1.47 (0.15) 
2008-2009 (Year 3) 199.2 19 201.3 18 0.70 (0.49) 


Results of the NJASK LAL analyses revealed no statistically significant effects of the whole-school 
intervention on student achievement on the NJASK in Year 1, Year 2, or Year 3 of the intervention 
(see Table 98). Due to the changes made to the NJASK, both treatment and comparison groups 
experienced a substantial drop in their test scores at the end of Year 1. In Year 3, both groups 
scored higher on the NJASK tests than they did in Year 1 and Year 2 of the study. Although the 
myriad limitations discussed earlier indicated the difficulty of finding an appropriate comparison 
group, the fact that no baseline difference existed between the two groups does bolster confidence 


in the models to a small degree. Full regression results are available in Appendix E. 


Table 98. NJASK Regression Analysis Results 


Effect Estimate Pr < |t] 
Baseline differences between groups 0.56 0.916 
Treatment Effect in Year 1 -3.4 0.27 
Treatment Effect in Year 2 -3.6 0.25 
Treatment Effect in Year 3 -1.8 0.56 


5.D Discussion and Conclusion 


Results from the analysis of whole-school intervention indicate no difference between the treatment 
and comparison groups on the state’s Language Arts Literacy exam at the end of Year 1, Year 2 and 
Year 3 of the grant. With increased teaching days added to the school calendar, there was a 
reduction in professional development days allotted in the district. With fewer professional 
development days, teachers did not have the same opportunity as in previous years to attend 


trainings. The low level of participation of teachers in both NJCU and NUA large-group trainings 


30Since the data was downloaded from the New Jersey State Department of Education website, in the baseline year, 3 schools in the treatment group 
and 6 schools in the comparison group did not have school level NJASK scores. In year 2, three schools in the comparison group had missing 
NJASK data and in year 3, one school in the comparison group did not have NJASK test score. 
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described in the previous chapter could potentially explain the lack of program impact. Whole- 
school intervention effects may not have been found because of low participation rates of teachers 
rather than the ineffectiveness of the intervention itself. High teacher turnover could also have 
affected the implementation of the literacy strategies that were the cornerstone of the whole-school 
intervention. Twenty percent of the teachers who participated in whole-school intervention left their 
school and did not continue with the program in Year 2 and Year 3. Since changing test scores rests 
on first changing the practices and behaviors of many teachers and administrators, future 
intervention programs should put strong emphasis on continued long term support for teacher 


professional development to ensure program participation and implementation fidelity. 
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Appendix A 


READ 180 Pacing Guide 


LV 


READ 180 MASTER LAYOUT 


6" Grade [Year 1] 


7 Grade [Year 2] 


8 Grade [Year 3] 


rBook 

Workshop 1 The New Americans 
Skill: Main Idea & Detail 

Writing Focus: Expository Writing 


tBook 

Workshop 7 Alien Invaders 

Skill: Cause and Effect 

Writing Focus: Persuasive Writing 


tBook Flex 

Workshop 4 Crime Lab Science 
Skill: Summarize 

Writing Focus: Expository Summary 


tBook 

Workshop 2 When Disaster Strikes 
Skill: Sequence of Events 

Writing Focus: Narrative Writing 


tBook 

Workshop 8 Turning Points 

Skill: Compare and Contrast 
Writing Focus: Descriptive Writing 


rBook Flex 

Workshop 5 Wired for Trouble 
Skill: Fact and Opinion 

Writing Focus: Persuasive Writing 


tBook 

Workshop 3 Identity Crisis 

Skill: Story Elements 

Writing Focus: Literature Response 


tBook 

Workshop 9 The Streets of Harlem 
Skill: Make Inferences 

Writing Focus: Personal Narrative 


tBook Flex 

Workshop 6 Facing the Elements 
Skill: Story Elements 

Writing Focus: Literature Review 


rBook 
Workshop 4 Stolen Childhoods 
Skill: Summarize 
Writing Focus: Expository Summary 
rBook 
Workshop 5 Under Pressure 
Skill: Problem and Solution 
Writing Focus: Persuasive Writing 


tBook Flex 
Workshop 1 Eyes on the Graduation Prize 
Skill: Main Idea & Detail 
Writing Focus: Expository Writing 
tBook Flex 
Workshop 2 Tsunami: Disaster of a Century 
Skill: Sequence of Events 
Writing Focus: Narrative Writing 


tBook Flex 
Workshop 7 Creatures of the Deep 
Skill: Cause and Effect 

Writing Focus: Descriptive Writing 
tBook Flex 
Workshop 8 Going Global 
Skill: Compare and Contrast 
Writing Focus: Persuasive Writing 


rBook 

Workshop 6 Poe: The Master of Horror 
Skill: Story Elements 

Writing Focus: Literature Review 


rBook Flex 

Workshop 3 Long Journey to Justice 
Skill: Story Elements 

Writing Focus: Literature Response 


tBook Flex 

Workshop 9 The Art of the Memoir 
Skill: Make Inferences 

Writing Focus: Personal Narrative 


Appendix B 


Targeted Intervention - 


Student Outcomes: HLM Output 
and 
Student Outcomes: Detailed Tables 


Program: HLM 6 Hierarchical Linear and Nonlinear Modeling 
Authors: Stephen Raudenbush, Tony Bryk, & Richard Congdon 
Publisher: Scientific Software International, Inc. (c) 20 
techsupport@ssicentral.com 
www.ssicentral.com 


Module: HLM2.EXE (6.06.2857.2) 


Date: 7 January 2011, Friday 
Time: 10:27: 5 


SPECIFICATIONS FOR THIS HLM2 RUN 


Problem Title: Vocab Overall 


Analysis\HLM\Data\MDM\Year 4\Treatl_ 678Grade\Overall.mdm 
The command file for this run = P:\Data Analysis\H 
4\Treatl_ 678Grade\Vocab Overall.hlm 


00 


The data source for this run = \\rk8\vol805\STRIVING\Data 


LM\Models\Year 


Output file name = \\rk8\vol805\STRIVING\Data Analysis\HI 
4\Treatl_ 678Grade\Vocab Overall.txt 


The maximum number of level-1 units = 2555 
The maximum number of level-2 units = 19 
The maximum number of iterations = 100 


Method of estimation: restricted maximum likelihood 


Weighting Specification 
Weight 

Variable 

Weighting? Name Normalized? 
Level 1 no 

Level 2 no 

Precision no 


a 


[The outcome variable is VOCAB 


[The model specified for the fixed effects was: 


ievel-1 Level-2 
Coefficients Predictors 


INTRCPT1, BO INTRCPT2, GOO 
EAT, G01 


R 

NELGIBLE, G02 
YRIMPROV, G03 
NE 
N 
i 


Lh, GOA 
SPECED, G05 
EANSCHO, G06 


B-1 


LM\Models\Year 


ENDER slope, 


P slope, B2 
ECED slope, 
MBLK slope, 


Bl INTRC 
INTRCPT2 
B3 INTRC 


PT2, 
, G20 
PT2, 


B4 INTRCPT2, 


G10 


G30 
G40 


MY1_7 
MY1 8 
MY2 6 
MY3 6 
MY4 6 
RENJS 


QAAARAADNEA 


AP AlP AP AP AP AP AP AP AP OP 


SCO 


Ue ed 


slope, 
slope, 
slope, 
slope, 
slope, 
slope, 


B5 
B6o 
B7 
B8 
B9 


INT 
NT 
NT 
NT 
NT 


I 
il 
I 
I 


RCPT2, 
RCPT2, 
RCPT2, 
RCPT2, 
RCPT2, 


B10 INTRCPT2, 


G50 
G60 
G70 
G80 
G90 
G100 


The residual parameter variance for this level-1 coefficient has 


been se 
to zero. 
'S' — This 
's' - This 


level-1l predictor has been centered around its grand mean. 
level-2 predictor has been centered around its grand mean. 


The model specified for the covariance components was: 


Sigma squared (constant across level-2 units) 


Tau dimensions 
INTRCPT1 


Summary of the model specified (in equation format) 
Level-1 Model 
Y = BO + B1* (GENDER) + B2* (LEP) + B3*(SPECED) + B4*(RDUMBLK) + 
B5* (GDUMY1_ 7) + B6* (GDUMY1 8) + B7* (GDUMY2_ 6) +  B8* (GDUMY3_ 6) + 
B9* (GDUMY4 6) + B10*(SCORENJS) + R 
Level-2 Model 
BO = GOO + GO1* (TREAT) + G02* (NELGIBLE) + G03* (YRIMPROV) + 
G0O4* (NELL) 
+ GO5* (NSPECED) + G0O6* (MEANSCHO) + UO 
Bl = G10 
B2 = G20 
B3 = G30 
B4 = G40 
B5 = G50 
B6 = G60 
B7 = G70 
B8 = G80 
B9 = G90 
B10 = G100 


Run-time deletion has reduced the number of level-1 records to 2256 


Iterations stopped due to small change in likelihood function 


B-2 


KKKKKKK ITERATION 6 KAKKKKKK 


Sigma_squared = 674.69478 


Tau 
INTRCPT1,BO 4.68928 


Tau (as correlations) 
INTRCPT1,BO 1.000 


INTRCPT1, BO 0.425 


The value of the likelihood function at iteration 6 = -1.053761E+004 
The outcome variable is VOCAB 


Final estimation of fixed effects: 


Standard Approx. 
Fixed Effect Coefficient Error T-ratio d.f. P-value 


For INTRCPT1, BO 

NTRCPT2, GOO 613.368326 1.198453 511.800 12 0.000 
REAT, GO1l 1.387250 1.698014 0.817 12 0.430 
ELGIBLE, GO2 -0.018829 0.044233 -0.426 12 0.677 
RIMPROV, GO3 0.187863 0.569715 0.330 12 0.747 
ELL, GO4 -0.013969 0.067520 -0.207 12 0.840 
PECED, GO5 0.041099 0.078739 0.522 12 0.611 
ANSCHO, G06 0.780565 0.358632 2.177 12 0.050 
GENDER slope, Bl 

PT2, G10 4.106570 1.126971 3.644 2239 0.001 
EP slope, B2 

PT2, G20 -2.561594 1.964320 -1.304 2239 0.193 
PECED slope, B3 

PT2, G30 -8.115935 1.230057 -6.598 2239 0.000 
RDUMBLK slope, B4 

RCPT2, G40 0.228849 1.419928 0.161 2239 0.872 
GDUMY1_7 slope, B5 

RCPT2, G50 23.293958 1.904809 12.229 2239 0.000 
GDUMY1 8 slope, B6 

RCPT2, G60 35.954048 1.870000 19.227 2239 0.000 
GDUMY2_ 6 slope, B7 

RCPT2, G70 -0.850606 1.878972 -0.453 2239 0.650 
GDUMY3 6 slope, B8 

RCPT2, G80 4.844106 1.856649 2.609 2239 0.009 
GDUMY4 6 slope, B9 

RCPT2, G90 4.877767 1.871098 2.607 2239 0.010 


on Nn 


KR 


Zogdg 

dw H 
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Qa 
Gl 


e) 
R 
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Zow 
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Zz 
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For SCORENJS slope, B10 
INTRCPT2, G100 7.187118 0.600289 11.973 2239 0.000 
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The outcome variable is VOCAB 


Final estimation of fixed effects 
(with robust standard errors) 


Standard Approx. 
Fixed Effect Coefficient Error T-ratio d.f. P-value 


For INTRCPT1, BO 

INTRCPT2, GOO 613.368326 0.978953 626.556 12 0.000 
TREAT, GO1 1.387250 1.331667 1.042 12 0.319 
NELGIBLE, G02 -0.018829 0.033595 -0.560 12 0.585 
YRIMPROV, G03 0.187863 0.419752 0.448 12 0.662 
NELL, G04 -0.013969 0.038524 -0.363 12 0.723 
NSPECED, G05 0.041099 0.052737 0.779 12 0.451 
MEANSCHO, G06 0.780565 0.208500 3.744 12 0.003 

For GENDER slope, Bl 

INTRCPT2, G10 4.106570 1.166469 3.521 2239 0.001 
For LEP slope, B2 

INTRCPT2, G20 -2.561594 1.238423 -2.068 2239 0.038 
For SPECED slope, B3 

INTRCPT2, G30 -8.115935 1.062454 -7.639 2239 0.000 
For RDUMBLK slope, B4 

INTRCPT2, G40 0.228849 1.083403 0.211 2239 0.833 
For GDUMY1_ 7 slope, B5 

INTRCPT2, G50 23.293958 1.793063 12.991 2239 0.000 
For GDUMY1 8 slope, Bé 

INTRCPT2, G60 35.954048 2.365950 15.196 2239 0.000 
For GDUMY2 6 slope, B7 

INTRCPT2, G70 -0.850606 1.681229 -0.506 2239 0.612 
For GDUMY3 6 slope, B8 

INTRCPT2, G80 4.844106 1.455078 3.329 2239 0.001 
For GDUMY4 6 slope, B9 

INTRCPT2, G90 4.877767 2.335847 2.088 2239 0.037 
For SCORENJS slope, B10 

INTRCPT2, G100 7.187118 0.871867 8.243 2239 0.000 


The robust standard errors are appropriate for datasets 


moderate to 


having 


large number of level 2 units. These data do not meet this criterion. 


Final estimation of variance components: 


Random Effect Standard Variance df Chi-square P-value 
Deviation Component 
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a 


INTRCPT1, UO 2.16547 4.68928 12 20.14840 0.064 
level-1, R 25.97489 674.69478 


Deviance = 21075.212396 


Number of estimated parameters 2 
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B1. 


Analysis Group 1— Vocabulary -- 1 year of treatment for 6th, 7th, & 8th 


graders 
combined 


Table 1. Vocabulary — Overall 


Fixed Effects 

Standard 
Effect Estimate Error DF t Value Pr > |t| 
Intercept 613.37 0.98 12 626.56 0.000 
Treatment (S) 1.39 1.33 12 1.04 0.319 
Num eligible students (S) -0.02 0.03 12 -0.56 0.585 
Yr in need of improvement (S) 0.19 0.42 12 0.45 0.662 
Num. ELL students (S) -0.01 0.04 12 -0.36 0.723 
Num. Special Education students (S) 0.04 0.05 12 0.78 0.451 
Mean score of schools (S) 0.78 0.21 12 3.74 0.003 
GENDER 4.11 1.17 2239 3.52 0.001 
English Language Learners -2.56 1.24 2239 -2.07 0.038 
Special Education student -8.12 1.06 2239 -7.64 0.000 
African American 0.23 1.08 2239 0.21 0.833 
Grade 7 Year 1 23.29 1.79 2239 12.99 0.000 
Grade 8 Year 1 35.95 2.37 2239 15.20 0.000 
Grade 6 Year 2 -0.85 1.68 2239 -0.51 0.612 
Grade 6 Year 3 4.84 1.46 2239 3.33 0.001 
Grade 6 Year 4 4.88 2.34 2239 2.09 0.037 
Baseline NJ score 7.19 0.87 2239 8.24 0.000 
Random Effects 
Variance Components Estimate ICC 
Level-2 Random Intercept School 4.69 0.007 
Level-1 Residual Student 674.69 
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Table 2. Vocabulary — Female 


Fixed Effects 

Standard 
Effect Estimate Error DF t Value Pr> |t| 
Intercept 612.44 165." 12. 37114 0.000 
Treatment (S) 1.38 2.54 12 0.53 0.609 
Num eligible students (S) -0.04 0.08 12 -0.48 0.638 
Yr in need of improvement (S) 0.17 0.94 12 0.18 0.862 
Num. ELL students (S) -0.02 0.08 12 -0.29 0.781 
Num. Special Education students (S) 0.08 0.13 12 0.61 0.554 
Mean score of schools (S) 0.74 0.41 12 1.81 0.094 
English Language Learners -2.31 1.68 945 -1.38 0.169 
Special Education student -7.52 1.83 945 -4.11 0.000 
African American 1.20 1.50 945 0.80 0.424 
Grade 7 Year 1 24.77 2.55" - 945 9.78 0.000 
Grade 8 Year 1 40.10 4.98 945 8.06 0.000 
Grade 6 Year 2 -0.18 1.76 945 -0.10 0.919 
Grade 6 Year 3 4.19 2.66 945 1.58 0.114 
Grade 6 Year 4 Bay 2.55 945 2.06 0.039 
Baseline NJ score 6.76 1.27 945 5.51 0.000 
Random Effects 
Variance Components Estimate ICC 
Level-2 Random Intercept School 22.61 0.033 
Level-1 Residual Student 655.91 
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Table 3. Vocabulary — Male 


Fixed Effects 

Standard 
Effect Estimate Error DF t Value Pr> |t 
Intercept 613.60 1.07 12 571.76 0.000 
Treatment (S) 2.32 1.25 12 1.85 0.089 
Num eligible students (S) -0.02 0.02 12 -0.85 0.413 
Yr in need of improvement (S) 0.02 0.43 12 0.04 0.970 
Num. ELL students (S) 0.04 0.04 12 1.06 0.311 
Num. Special Education students (S) 0.01 0.04 12 0.37 0.719 
Mean score of schools (S) 0.32 0.20 12 1.57 0.142 
English Language Learners -1.91 1.75 1279 -1.10 0.274 
Special Education student -8.78 1.72 1279 -5.10 0.000 
African American -0.54 1.67 1279 -0.33 0.745 
Grade 7 Year 1 21.49 2.31 1279 9.32 0.000 
Grade 8 Year 1 32.43 2.11 1279 15.35 0.000 
Grade 6 Year 2 -1.99 2.27 1279 -0.88 0.381 
Grade 6 Year 3 4.57 2.21 1279 2.06 0.039 
Grade 6 Year 4 4.38 2.55 1279 1.72 0.086 
Baseline NJ score 7.58 0.82 1279 9.21 0.000 
Random Effects 
Variance Components Estimate ICC 
Level-2 Random Intercept School 0.64 0.001 
Level-1 Residual Student 687.91 
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Table 4. Vocabulary - African American 


Fixed Effects 

Standard 
Effect Estimate Error DF t Value Pr> |t| 
Intercept 614.22 1.34 12 458.18 0.000 
Treatment (S) 1.30 1.83 12 0.71 0.492 
Num eligible students (S) -0.01 0.05 12 -0.30 0.768 
Yr in need of improvement (S) 0.40 0.58 12 0.68 0.507 
Num. ELL students (S) 0.03 0.06 12 0.51 0.616 
Num. Special Education students (S) 0.00 0.07 12 -0.05 0.958 
Mean score of schools (S) 0.52 0.23 12 2.21 0.047 
GENDER 2.99 1.42 1251 2.10 0.035 
English Language Learners -1.82 4.31 1251 -0.42 0.672 
Special Education student -8.87 1.29 1251 -6.88 0.000 
Grade 7 Year 1 23.67 2.81 1251 8.41 0.000 
Grade 8 Year 1 34.81 2.62 1251 13.28 0.000 
Grade 6 Year 2 -0.57 2.34 1251 -0,.24 0.808 
Grade 6 Year 3 3.82 2.18 1251 1.75 0.080 
Grade 6 Year 4 5.61 3.07 1251 1.83 0.067 
Baseline NJ score 8.11 1.12 1251 7.22 0.000 
Random Effects 
Variance Components Estimate ICC 
Level-2 Random Intercept School 8.16 0.012 
Level-1 Residual Student 692.26 
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Table 5. Vocabulary — Hispanic 


Fixed Effects 

Standard 
Effect Estimate Error DF t Value Pr> |t 
Intercept 612.51 1.63 11 374.91 0.000 
Treatment (S) 2.81 1.84 11 1.53 0.154 
Num eligible students (S) 0.01 0.04 11 0.25 0.809 
Yr in need of improvement (S) -0.51 0.41 11 -1.24 0.242 
Num. ELL students (S) -0.09 0.07 11 -1.34 0.209 
Num. Special Education students (S) 0.00 0.07 11 0.02 0.981 
Mean score of schools (S) -0.06 0.30 11 -0.20 0.848 
GENDER 5.55 1.71 929 3.24 0.002 
English Language Learners -1.45 1.08 929 -1.35 0.177 
Special Education student -6.99 1.32 929 -5.29 0.000 
Grade 7 Year 1 22.88 2.26 929 10.12 0.000 
Grade 8 Year 1 37.56 3.76 929 10.00 0.000 
Grade 6 Year 2 -1.86 2.51 929 -0.74 0.458 
Grade 6 Year 3 4.92 1.87 929 2.64 0.009 
Grade 6 Year 4 3.33 3.23 929 1.03 0.303 
Baseline NJ score 6.45 0.89 929 7.24 0.000 
Random Effects 
Variance Components Estimate ICC 
Level-2 Random Intercept School 5.97 0.009 
Level-1 Residual Student 657.16 


Table 6. Vocabulary - Special Education 


Fixed Effects 

Standard 
Effect Estimate Error DF t Value Pr> |t| 
Intercept 604.11 1.52 12 396.52 0.000 
Treatment (S) 4.03 1.82 12 2.21 0.047 
Num eligible students (S) -0.01 0.05 12 -0.13 0.897 
Yr in need of improvement (S) 0.03 0.59 12 0.05 0.964 
Num. ELL students (S) 0.05 0.05 12 0.95 0.363 
Num. Special Education students (S) -0.05 0.07 12 -0.67 0.513 
Mean score of schools (S) 0.02 0.21 12 0.10 0.925 
GENDER 2.95 1.88 882 1.57 0.117 
English Language Learners -2.86 2.23 882 -1.28 0.200 
African American -2.20 1.41 882 -1.56 0.119 
Grade 7 Year 1 29.66 3.19 882 9.30 0.000 
Grade 8 Year 1 41.53 4.61 882 9.02 0.000 
Grade 6 Year 2 1.22 2.83 882 0.43 0.666 
Grade 6 Year 3 8.34 3.39 882 2.46 0.014 
Grade 6 Year 4 7.36 3.14 882 2.34 0.019 
Baseline NJ score 6.94 1.01 882 6.85 0.000 
Random Effects 
Variance Components Estimate ICC 
Level-2 Random Intercept School 5.65 0.008 
Level-1 Residual Student 718.39 
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B2. Analysis Group 2 — Vocabulary -- 2 years of treatment for 7th & 8th 


graders combined 


Table 7. Vocabulary —Overall 


Fixed Effects 

Standard 
Effect Estimate Error DF t Value Pr> |t| 
Intercept 628.20 0.81 12. = 773.72 0.000 
Treatment (S) 1.63 1.15 12 1.41 0.183 
Num eligible students (S) -0.02 0.04 12 -0.55 0.589 
Yr in need of improvement (S) -0.57 0.33 12 -1.73 0.109 
Num. ELL students (S) 0.09 0.04 12 2.06 0.061 
Num. Special Education students (S) -0.01 0.06 12 -0.20 0.845 
Mean score of schools (S) -0.04 0.17 12 -0.23 0.826 
GENDER 2.25 1.35 1392 1.67 0.095 
English Language Learners -2.44 2.18 1392 -1.12 0.262 
Special Education student -5.69 1.51 1392 -3.76 0.000 
African American 2.53 1.61 1392 1.58 0.115 
Grade 8 Year 2 17.11 2.51 1392 6.81 0.000 
Grade 7 Year 3 3.14 2.12 1392 1.48 0.140 
Grade 7 Year 4 1.26 2.09 1392 0.60 0.546 
Baseline NJ score 10.50 1.05 1392 10.01 0.000 
Random Effects 
Variance Components Estimate ICC 
Level-2 Random Intercept School 1.01 0.002 
Level-1 Residual Student 538.81 
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Table 8. Vocabulary — Female 


Fixed Effects 

Standard 
Effect Estimate Error DF t Value Pr> |t| 
Intercept 630.00 0.60 12 1050.74 — 0.000 
Treatment (S) 0.64 0.86 12 0.74 0.473 
Num eligible students (S) -0.03 0.02 12 -1.32 0.211 
Yr in need of improvement (S) -0.30 0.30 12 -1.01 0.334 
Num. ELL students (S) 0.00 0.02 12 0.13 0.897 
Num. Special Education students (S) 0.01 0.03 12 0.23 0.821 
Mean score of schools (S) 0.34 0.11 = 12 3.08 0.010 
English Language Learners 0.21 2.66 599 0.08 0.936 
Special Education student -4.58 ZAG, 599 -2.18 0.029 
African American 1,13 218: 599 0.52 0.603 
Grade 8 Year 2 14.24 4.13 599 3.45 0.001 
Grade 7 Year 3 1.88 Zito 3999 0.68 0.495 
Grade 7 Year 4 0.40 2.64 599 0.15 0.879 
Baseline NJ score 9.03 Lei 509 7.58 0.000 
Random Effects 
Variance Components Estimate ICC 
Level-2 Random Intercept School 0.11 0.000 
Level-1 Residual Student 476.12 
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Table 9. Vocabulary — Male 


Fixed Effects 

Standard 
Effect Estimate Error DF_ tValue Pr> |t| 
Intercept 626.69 1.40 12 449.08 0.000 
Treatment (S) 2.88 2.05 12 1.41 0.185 
Num eligible students (S) -0.02 0.06 12 -0.31 0.764 
Yr in need of improvement (S) -0.89 0.64 12 -1.39 0.189 
Num. ELL students (S) 0.13 0.07 12 1.83 0.092 
Num. Special Education students (S) -0.03 0.10 12 -0.31 0.761 
Mean score of schools (S) -0.14 0.21 12 -0.68 0.507 
English Language Learners -5.83 3.49 780 -1.67 0.094 
Special Education student -6.11 1.67 780 -3.65 0.001 
African American 3.46 2.47 780 1.40 0.162 
Grade 8 Year 2 19.14 2.47 780 7.74 0.000 
Grade 7 Year 3 4.01 3.14 780 1.28 0.203 
Grade 7 Year 4 2.66 2.48 780 1.07 0.285 
Baseline NJ score 11.66 1.29 780 9.01 0.000 
Random Effects 
Variance Components Estimate ICC 
Level-2 Random Intercept School 44.97 0.020 
Level-1 Residual Student 579.58 
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Table 10. Vocabulary - African American 


Fixed Effects 

Standard 
Effect Estimate Error DF t Value Pr > |t| 
Intercept 629.77 0.96 12 654.47 0.000 
Treatment (S) 1.30 1.28 12 1.02 0.329 
Num eligible students (S) -0.01 0.03 12 -0.27 0.794 
Yr in need of improvement (S) -0.31 0.50 12 -0.63 0.541 
Num. ELL students (S) 0.13 0.07 12 2.04 0.064 
Num. Special Education students (S) -0.07 0.04 12 -1.74 0.107 
Mean score of schools (S) -0.04 0.10 12 -0.40 0.699 
GENDER 3.21 1.56 745 2.06 0.039 
English Language Learners -4.03 3.94 745 -1.02 0.307 
Special Education student -6.17 1.68 745 -3.67 0.000 
Grade 8 Year 2 19.93 2.42 745 8.25 0.000 
Grade 7 Year 3 5.80 2.36 745 2.46 0.014 
Grade 7 Year 4 2.92 2.46 745 1.18 0.237 
Baseline NJ score 9.93 0.82 745 12.16 0.000 
Random Effects 
Variance Components Estimate ICC 
Level-2 Random Intercept School 0.55 0.001 
Level-1 Residual Student 527.85 
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Table 11. Hispanic 


Fixed Effects 

Standard 
Effect Estimate Error DF_t Value Pr> |t| 
Intercept 625.89 2.51 10 = 249.35 0.000 
Treatment (S) 5.00 3.78 10 1.32 0.216 
Num eligible students (S) -0.15 0.10 10 -1.50 0.163 
Yr in need of improvement (S) 0.29 0.91 10 0.32 0.754 
Num. ELL students (S) -0.10 0.12 10 -0.81 0.438 
Num. Special Education students (S) 0.34 0.19 10 1.75 0.110 
Mean score of schools (S) 0.30 0.18 10 1.67 0.125 
GENDER 0.54 1.97 599 0.28 0.783 
English Language Learners -0.79 2.15 599 -0.37 0.713 
Special Education student -4.66 2.04 599 -2,28 0.023 
Grade 8 Year 2 13.60 3.30 599 4.12 0.000 
Grade 7 Year 3 0.05 2.44 599 0.02 0.985 
Grade 7 Year 4 0.22 2.35 599 0.09 0.927 
Baseline NJ score 11.32 1.64 599 6.90 0.000 
Random Effects 
Variance Components Estimate ICC 
Level-2 Random Intercept School 35.12 0.059 
Level-1 Residual Student 556.71 


Table 12. Vocabulary - Special Education 


Fixed Effects 

Standard 
Effect Estimate —_ Error DF t Value Pr> |t| 
Intercept 619.98 1.18 12 525.45 0.000 
Treatment (S) 2.24 1.81 12 1.24 0.240 
Num eligible students (S) -0.05 0.05 12 -0.96 0.356 
Yr in need of improvement (S) -0.82 0.44 12 -1.86 0.086 
Num. ELL students (S) 0.15 0.06 12 2.73 0.019 
Num. Special Education students (S) 0.02 0.07 12 0.25 0.806 
Mean score of schools (S) -0.11 0.15 12 -0.71 0.489 
GENDER 0.66 1.73 581 0.38 0.701 
English Language Learners 0.81 4.75 581 0.17 0.864 
African American 2.93 2.45 581 1.20 0.233 
Grade 8 Year 2 18.57 2.91 581 6.37 0.000 
Grade 7 Year 3 4.64 2.15 581 2.16 0.031 
Grade 7 Year 4 1.08 2.71 581 0.40 0.690 
Baseline NJ score 11.17 1.04 581 10.73 0.000 
Random Effects 
Variance Components Estimate ICC 
Level-2 Random Intercept School 52) 0.009 
Level-1 Residual Student 554.19 
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B3. Analysis Group 3 — Vocabulary -- 3 years of treatment for 8th graders 


Table 13. Vocabulary — Overall 


Fixed Effects 
Standard 

Effect Estimate Error DF t Value Pr> |t| 
Intercept 641.47 1.46 12 439.64 0.000 
Treatment (S) 1.44 2.12 12 0.68 0.509 
Num eligible students (S) -0.08 0.04 12 -2.01 0.067 
Yr in need of improvement (S) -0.06 0.77 12 -0.08 0.938 
Num. ELL students (S) 0.13 0.08 12 1.64 0.127 
Num. Special Education students (S) 0.05 0.09 12 0.62 0.548 
Mean score of schools (S) -0.06 0.26 12 -0.25 0.806 
GENDER 3.51 1.82 920 1.92 0.054 
English Language Learners -4.67 2.42 920 -1.93 0.053 
Special Education student -6.04 1.59 920 -3.79 0.000 
Aftican American -0.42 2.52 920 -0.17 0.870 
Grade 8 Year 4 -1.30 2.22 920 -0.59 0.557 
Grade 8 Year 5 1.37 2.66 920 0.51 0.607 
Baseline NJ score 10.95 1.60 920 6.86 0.000 

Random Effects 

Variance Components Estimate ICC 

Level-2 Random Intercept School 22.32 0.038 

Level-1 Residual Student 572.77 
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Table 14. Vocabulary — Female 


Fixed Effects 
Standard 

Effect Estimate Error DF t Value Pr > |t| 
Intercept 641.87 1.16 12 551.76 0.000 
Treatment (S) 0.24 237 12 0.10 0.922 
Num eligible students (S) 0.01 0.05 12 0.19 0.851 
Yr in need of improvement (S) 0.34 0.70 12 0.48 0.637 
Num. ELL students (S) 0.09 0.11 12 0.81 0.435 
Num. Special Education students (S) -0.09 0.09 12 -0.98 0.345 
Mean score of schools (S) 0.02 0.29 12 0.07 0.948 
English Language Learners -1.29 4.85 385 -0.27 0.791 
Special Education student -10.44 2.62 385 -3.99 0.000 
African American 3.67 2.91 385 1.26 0.209 
Grade 8 Year 4 -0.98 2.19 385 -0.45 0.653 
Grade 8 Year 5 2.47 3.25 385 0.76 0.447 
Baseline NJ score 8.83 1.52 385 5.83 0.000 

Random Effects 

Variance Components Estimate ICC 

Level-2 Random Intercept School 19.36 0.042 19.35672 

Level-1 Residual Student 444.39 444.38914 
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Table 15. Vocabulary — Male 
Fixed Effects 
Standard 

Effect Estimate Error DF t Value Pr> |t| 
Intercept 641.01 2.04 12 314.38 0.000 
Treatment (S) 2.74 2.76 12 0.99 0.342 
Num eligible students (S) -0.14 0.06 12 -2.41 0.033 
Yr in need of improvement (S) -0.42 0.98 12 -0.44 0.671 
Num. ELL students (S) 0.11 0.11 12 0.95 0.359 
Num. Special Education students (S) 0.18 0.12 12 1.57 0.143 
Mean score of schools (S) 0.06 0.30 12 0.20 0.844 
English Language Learners -7.23 2.71 523 -2.67 0.008 
Special Education student -2.94 2.41 523 -1.22 0.223 
African American -2.61 2.60 523 -1.00 0.316 
Grade 8 Year 4 -1.56 3.49 523 -0.45 0.655 
Grade 8 Year 5 1.07 3.25 523 0.33 0.742 
Baseline NJ score 12.17 1.81 523 6.73 0.000 

Random Effects 

Variance Components Estimate ICC 

Level-2 Random Intercept School 29.38 0.043 29.37776 

Level-1 Residual Student 656.86 656.85956 


Table 16. Vocabulary — African American 


Fixed Effects 
Standard 

Effect Estimate Error DF t Value Pr> |t| 
Intercept 640.49 912 1B 30278 > 20000 
Treatment (S) 1.46 2.67 12 0,55 0.594 
Num eligible students (S) -0.07 0.04 12 -1.67 0.121 
Yr in need of improvement (S) 0.03 1.04 12 0.03 0.980 
Num. ELL students (S) 0.33 0.12 12 2.68 0.020 
Num. Special Education students (S) 0.05 0.09 12 0.51 0.617 
Mean score of schools (S) -0.34 0.32 12 -1.05 0.315 
GENDER 1.00 2.62 496 0.38 0.701 
Special Education student -8.86 2.06 496 -4,30 0.000 
Grade 7 Year 4 -0.89 1.80 496 -0.50 0.618 
Grade 8 Year 5 0.42 2.55 496 0.17 0.869 
Baseline NJ score 11.43 1.57 496 7.28 0.000 

Random Effects 

Variance Components Estimate ICC 

Level-2 Random Intercept School 22.32 0.038 22.31894 

Level-1 Residual Student 572.32 572.31724 
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Table 17. Vocabulary — Hispanic 


Fixed Effects 
Standard 
Effect Estimate Error DF t Value Pr> |t| 
Intercept 646.60 4.09 10 = 158.27 0.000 
Treatment (S) -0.73 5.22 10 -0.14 0.892 
Num eligible students (S) -0.23 0.11 10 -2.06 0.066 
Yr in need of improvement (S) 0.86 1.12 10 0.76 0.462 
Num. ELL students (S) 0.08 0.21 10 0.39 0.707 
Num. Special Education students (S) 0.27 0.17 10 1.58 0.146 
Mean score of schools (S) -0.13 0.26 10 -0.50 0.627 
GENDER 6.11 1.57 392 3.90 0.000 
English Language Learners -2.90 2:31 392 -1.25 0.211 
Special Education student -2.28 2.45 392 -0.93 0.353 
Grade 8 Year 4 -2.26 3.96 392 -0.57 0.569 
Grade 8 Year 5 0.86 3.67 392 0.23 0.816 
Baseline NJ score 10.97 2.58 392 4.25 0.000 
Random Effects 
Variance Components Estimate ICC 
Level-2 Random Intercept School 66.13 0.105 
Level-1 Residual Student 563.05 
Table 18. Vocabulary — Special Education 
Fixed Effects 
Standard 
Effect Estimate Error DF tValue Pr> |t| 
Intercept 635.89 2.33 12 272.45 0.000 
Treatment (S) -1.31 2.35 12 -0.56 0.587 
Num eligible students (S) -0.07 0.05 12 -1.24 0.241 
Yr in need of improvement (S) 0.43 0.81 12 0.54 0.602 
Num. ELL students (S) 0.08 0.09 12 0.85 0.413 
Num. Special Education students (S) 0.07 0.11 12 0.62 0.547 
Mean score of schools (S) 0.17 0.23. 12 0.73 0.477 
GENDER 7.51 2.09 383 3.59 0.001 
English Language Learners -6.14 3.97 383 -1.55 0.122 
African American -3.47 2.61 383 -1.33 0.184 
Grade 8 Year 4 0.41 3.35 383 0.12 0.903 
Grade 8 Year 5 0.70 4.03 383 0.17 0.862 
Baseline NJ score 11.29 1.78 383 6.36 0.000 
Random Effects 
Variance Components Estimate ICC 
Level-2 Random Intercept School 9.41 0.017 
Level-1 Residual Student 542.62 
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B4. Analysis Group 1— Comprehension -- 1 year of treatment for 6th, 7th, 
& 8th graders combined 


Table 19. Comprehension — Overall 


Fixed Effects 
Standard 

Effect Estimate Error DF t Value Pr > |t| 
Intercept 609.11 0.87 12. 697.95 0.000 
Treatment (S) 113 1.14 12 0.99 0.344 
Num eligible students (S) -0.09 0.02 12 -3.69 0.004 
Yr in need of improvement (S) 0.20 0.35 12 0.56 0.588 
Num. ELL students (S) 0.00 0.04 12 0.09 0.931 
Num. Special Education students (S) 0.18 0.03 12 5.35 0.000 
Mean score of schools (S) 0.80 0.23 12 3.49 0.005 
GENDER -2.95 0.98 2335 -3.01 0.003 
English Language Learners -0.75 1.03 2335 -0.74 0.462 
Special Education student -9.71 0.94 2335  -10.28 0.000 
African American -2.50 1.03 2335 -2.44 0.015 
Grade 7 Year 1 16.46 2.57 2335 6.39 0.000 
Grade 8 Year 1 36.06 1.62 2335 22.32 0.000 
Grade 6 Year 2 1.54 212 2335 0.73 0.468 
Grade 6 Year 3 5.82 2.35 2335 2.48 0.013 
Grade 6 Year 4 4.61 3.13 2335 1.47 0.141 
Baseline NJ score 6.23 0.72 2335 8.69 0.000 

Random Effects 

Variance Components Estimate ICC 

Level-2 Random Intercept School 4.68 0.009 

Level-1 Residual Student 536.67 
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Table 20. Comprehension — Female 


Fixed Effects 
Standard 

Effect Estimate Error DF_t Value Pr> |t| 
Intercept 612.05 0.93 12. 661.59 0.000 
Treatment (S) 1.94 1.40 12 1.39 0.190 
Num eligible students (S) -0.15 0.04 12 -4.19 0.001 
Yr in need of improvement (S) 0.07 0.38 12 0.19 0.857 
Num. ELL students (S) 0.07 0.04 12 1.71 0.113 
Num. Special Education students (S) 0.23 0.05 12 4,15 0.001 
Mean score of schools (S) 0.95 0.23 12 4.08 0.002 
English Language Learners 1.02 1.13 979 0.90 0.369 
Special Education student -11.47 1.51 979 -7.61 0.000 
African American -1.00 1.41 979 -0.71 0.478 
Grade 7 Year 1 16.09 2.95 979 5.46 0.000 
Grade 8 Year 1 35:72 2.64 979 13.54 0.000 
Grade 6 Year 2 0.12 3.55 979 0.03 0.974 
Grade 6 Year 3 5.70 3.98 979 1.43 0.153 
Grade 6 Year 4 5.41 4.79 979 1.13 0.260 
Baseline NJ score 595 1.09 979 5.44 0.000 

Random Effects 

Variance Components Estimate ICC 

Level-2 Random Intercept School 0.42 0.001 

Level-1 Residual Student 514.73 
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Table 21. Comprehension — Male 


Fixed Effects 
Standard 

Effect Estimate Error DF _t Value Pr > |t| 
Intercept 606.83 1.21 12 499.90 0.000 
Treatment (S) 1.10 1.45 12 0.76 0.462 
Num eligible students (S) -0.05 0.03 12 -1.79 0.098 
Yr in need of improvement (S) 0.03 0.47 12 0.07 0.948 
Num. ELL students (S) -0.04 0.06 12 -0.68 0.509 
Num. Special Education students (S) 0.14 0.05 12 2.96 0.013 
Mean score of schools (S) 0.46 0.20 12 2.26 0.043 
English Language Learners -2.67 1.81 1341 -1.47 0.141 
Special Education student -8.54 1.04 1341 -8.24 0.000 
Aftican American -3.91 1.86 1341 -2.10 0.035 
Grade 7 Year 1 16.84 3.50 1341 4.81 0.000 
Grade 8 Year 1 36.38 2.51 1341 14.47 0.000 
Grade 6 Year 2 2.30 2.70 1341 0.85 0.395 
Grade 6 Year 3 5.61 2.41 1341 2.33 0.020 
Grade 6 Year 4 4.23 2.87 1341 1.47 0.141 
Baseline NJ score 6.37 0.71 1341 8.97 0.000 

Random Effects 

Variance Components Estimate ICC 

Level-2 Random Intercept School 7.26 0.013 

Level-1 Residual Student 554.52 
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Table 22. Comprehension — African American 


Fixed Effects 
Standard 

Effect Estimate Error DF t Value Pr> |t| 
Intercept 607.77 1.20 12 506.26 0.000 
Treatment (S) 2.48 2.23 12 1.11 0.287 
Num eligible students (S) -0.08 0.05 12 -1.71 0.113 
Yr in need of improvement (S) -0.10 0.61 12 -0.16 0.878 
Num. ELL students (S) 0.05 0.07 12 0.72 0.488 
Num. Special Education students (S) 0.13 0.05 12 2.47 0.030 
Mean score of schools (S) 0.23 0.45 12 0.51 0.622 
GENDER -4.17 1.07 1314 -3,89 0.000 
English Language Learners -10.10 11.15 1314 -0.91 0.366 
Special Education student -9.04 1.51 1314 -6.01 0.000 
Grade 7 Year 1 16.38 3.36 1314 4.88 0.000 
Grade 8 Year 1 35.26 2.11 1314 16.74 0.000 
Grade 6 Year 2 2.01 2.67 1314 0.75 0.453 
Grade 6 Year 3 4.46 3.31 1314 1.35 0.178 
Grade 6 Year 4 6.03 2.64 1314 2.28 0.023 
Baseline NJ score 6.84 0.75 1314 9.13 0.000 

Random Effects 

Variance Components Estimate ICC 

Level-2 Random Intercept School 21:31 0.041 

Level-1 Residual Student 501.15 
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Table 23. Comprehension — Hispanic 


Fixed Effects 
Standard 

Effect Estimate Error DF t Value Pr> |t| 
Intercept 611.53 1.52 11 402.48 0.000 
Treatment (S) 1.10 1.62 11 0.68 0.510 
Num eligible students (S) -0.12 0.03 11 -3.85 0.003 
Yr in need of improvement (S) 0.01 0.35 11 0.02 0.983 
Num. ELL students (S) -0.09 0.05 11 -1.86 0.090 
Num. Special Education students (S) 0.25 0.05 11 4.54 0.001 
Mean score of schools (S) 0.13 0.18 11 0.69 0.503 
GENDER -1.86 1.93 960 -0.97 0.335 
English Language Learners 0.64 0.98 960 0.65 0.516 
Special Education student -10.41 1.80 960 -5.78 0.000 
Grade 7 Year 1 16.83 2.75 960 6.13 0.000 
Grade 8 Year 1 37.05 2.49 960 14.91 0.000 
Grade 6 Year 2 0.66 3.07 960 0.22 0.829 
Grade 6 Year 3 6.35 2.65 960 2.39 0.017 
Grade 6 Year 4 1.67 4.72 960 0.35 0.724 
Baseline NJ score 5.73 0.97 960 5.91 0.000 

Random Effects 

Variance Components Estimate ICC 

Level-2 Random Intercept School 0.79 0.001 

Level-1 Residual Student 583.31 
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Table 24. Comprehension — Special Education 


Fixed Effects 
Standard 

Effect Estimate Error DF t Value Pr > |t| 
Intercept 600.19 1.19 12 506.37 0.000 
Treatment (S) 1.32 1.57 12 0.84 0.417 
Num eligible students (S) -0.05 0.02 12 -2.08 0.059 
Yr in need of improvement (S) 0.01 0.43 12 0.03 0.981 
Num. ELL students (S) 0.06 0.04 12 1.62 0.131 
Num. Special Education students (S) 0.06 0.04 12 1.43 0.178 
Mean score of schools (S) -0.08 0.14 12 -0.62 0.547 
GENDER -0.64 1.31 906 -0.49 0.624 
English Language Learners 4.58 2.24 906 2.05 0.041 
African American -0.94 1.76 906 -0.53 0.594 
Grade 7 Year 1 13.33 3.31 906 4.03 0.000 
Grade 8 Year 1 32.94 1.95 906 16.87 0.000 
Grade 6 Year 2 -3.05 2.20 906 -1.39 0.165 
Grade 6 Year 3 1.05 2.81 906 0.38 0.708 
Grade 6 Year 4 -1.94 3.15 906 -0.62 0.537 
Baseline NJ score 5.66 0.81 906 7.00 0.000 

Random Effects 

Variance Components Estimate ICC 

Level-2 Random Intercept School 1.37 0.003 

Level-1 Residual Student 533.80 
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B5. Analysis Group 2 —- Comprehension -- 2 years of treatment for 7th & 


8th graders combined 


Table 25. Comprehension — Overall 


Fixed Effects 
Standard 

Effect Estimate Error DF t Value Pr> |t| 
Intercept 620.85 0.83 12 751.49 0.000 
Treatment (S) 3.59 1.33 12 2.71 0.019 
Num eligible students (S) -0.06 0.02 12 -2.51 0.028 
Yr in need of improvement (S) -0.45 0.33 12 -1.36 0.201 
Num. ELL students (S) 0.05 0.05 12 1.14 0.276 
Num. Special Education students (S) 0.09 0.07 12 1.31 0.215 
Mean score of schools (S) 0.14 0.15 12 0.99 0.343 
GENDER -2.36 1.01 1403 -2.35 0.019 
English Language Learners -2.79 1.84 1403 -1.52 0.129 
Special Education student -6.09 1.53 1403 -3.98 0.000 
Aftican American 0.90 1.05 1403 0.86 0.393 
Grade 8 Year 2 21.40 2.09 1403 10.26 0.000 
Grade 7 Year 3 -0.51 2.27 1403 -0.23 0.821 
Grade 7 Year 4 1.32 1.80 1403 0.74 0.462 
Baseline NJ score 9.42 0.95 1403 9.93 0.000 

Random Effects 

Variance Components Estimate ICC 

Level-2 Random Intercept School 7.38 0.016 

Level-1 Residual Student 441.69 
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Table 26. Comprehension — Female 


Fixed Effects 
Standard 

Effect Estimate Error DF t Value Pr> |t| 
Intercept 625.73 0.87 12 719.92 0.000 
Treatment (S) 1.08 1.45 12 0.75 0.469 
Num eligible students (S) -0.03 0.02 12 -1.23 0.243 
Yr in need of improvement (S) -0.60 0.40 12 -1.50 0.159 
Num. ELL students (S) 0.05 0.03 12 1.47 0.166 
Num. Special Education students (S) 0.00 0.06 12 -0.04 0.970 
Mean score of schools (S) 0.21 0.14 12 1.54 0.149 
English Language Learners -6.11 3.59 603 -1.70 0.088 
Special Education student -7.87 1.74 603 -4,52 0.000 
Aftican American -0.26 1.57 603 -0.17 0.870 
Grade 8 Year 2 19.17 2.43 603 7.88 0.000 
Grade 7 Year 3 -4,20 1.78 603 -2.36 0.019 
Grade 7 Year 4 -1.77 1.95 603 -0.91 0.365 
Baseline NJ score 9.30 1.16 603 8.05 0.000 

Random Effects 

Variance Components Estimate ICC 

Level-2 Random Intercept School 4.70 0.010 

Level-1 Residual Student 449.71 

Table 27. Comprehension — Male 
Fixed Effects 
Standard 

Effect Estimate Error DF t Value Pr> |t| 
Intercept 617.19 1.15 12 537.68 0.000 
Treatment (S) 5.21 1.47 12 3.56 0.004 
Num eligible students (S) -0.08 0.03 12 -2.70 0.020 
Yr in need of improvement (S) -0.66 0.42 12 -1.58 0.140 
Num. ELL students (S) 0.06 0.05 12 1.21 0.249 
Num. Special Education students (S) 0.14 0.08 12 1.80 0.096 
Mean score of schools (S) -0.02 0.16 12 -0.14 0.891 
English Language Learners 0.60 2.28 787 0.26 0.792 
Special Education student -5.00 1.77 787 -2.83 0.005 
African American 1.78 1.63 787 1.09 0.275 
Grade 8 Year 2 23.36 2.68 787 8.72 0.000 
Grade 7 Year 3 2.44 3.61 787 0.68 0.500 
Grade 7 Year 4 3.82 2.65 787 1.44 0.150 
Baseline NJ score 9.68 1.20 787 8.06 0.000 
Random Effects 

Variance Components Estimate ICC 

Level-2 Random Intercept School 5.88 0.013 

Level-1 Residual Student 435.67 
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Table 28. Comprehension — African American 


Fixed Effects 
Standard 

Effect Estimate Error DF t Value Pr> |t| 
Intercept 621.30 1.24 12 502.44 0.000 
Treatment (S) 3.99 1.78 12 2.23 0.045 
Num eligible students (S) -0.06 0.03 12 -2.05 0.063 
Yr in need of improvement (S) -0.64 0.51 12 -1.26 0.232 
Num. ELL students (S) 0.06 0.06 12 1.06 0.311 
Num. Special Education students (S) 0.09 0.07 12 125 0.237 
Mean score of schools (S) 0.23 0.12 12 1.90 0.081 
GENDER -1.89 1.20 753 -1.57 0.116 
English Language Learners -5.26 3.40 753 -1.55 0.122 
Special Education student -7.81 1.65 753 -4.74 0.000 
Grade 8 Year 2 22.81 2.59 753 8.81 0.000 
Grade 7 Year 3 0.22 2.59 753 0.09 0.933 
Grade 7 Year 4 1.19 2.52 753 0.47 0.638 
Baseline NJ score 8.89 0.88 753 10.05 0.000 
Random Effects 

Variance Components Estimate ICC 

Level-2 Random Intercept School 13.56 0.031 

Level-1 Residual Student 420.70 

Table 29. Comprehension — Hispanic 
Fixed Effects 
Standard 

Effect Estimate Error DF t Value Pr> |t| 
Intercept 621.54 1.50 10 413.32 0.000 
Treatment (S) 1.90 1.90 10 1.00 0.341 
Num eligible students (S) -0.06 0.04 10 -1.48 0.169 
Yr in need of improvement (S) -0.36 0.35 10 -1.04 0.323 
Num. ELL students (S) 0.00 0.08 10 0.01 0.992 
Num. Special Education students (S) 0.12 0.07 10 1.69 0.121 
Mean score of schools (S) 0.04 0.11 10 0.38 0.714 
GENDER -4.10 1.53 602 -2.67 0.008 
English Language Learners -2.37 1.84 602 -1.29 0.199 
Special Education student -4.40 2.09 602 -2.11 0.035 
Grade 8 Year 2 19.68 2.81 602 6.99 0.000 
Grade 7 Year 3 -0.29 3.56 602 -0.08 0.935 
Grade 7 Year 4 0.83 2.04 602 0.41 0.683 
Baseline NJ score 9.54 1.58 602 6.02 0.000 

Random Effects 

Variance Components Estimate ICC 

Level-2 Random Intercept School 0.17 0.000 

Level-1 Residual Student 466.74 


Table 30. Comprehension — Special Education 


Fixed Effects 
Standard 

Effect Estimate Error DF t Value Pr> |t| 
Intercept 611.20 1.33 12 458.99 0.000 
Treatment (S) 5.52 1.70 12 3.26 0.007 
Num eligible students (S) -0.16 0.04 12 -4.61 0.001 
Yr in need of improvement (S) -0.25 0.39 12 -0.63 0.538 
Num. ELL students (S) 0.23 0.05 12 4.61 0.001 
Num. Special Education students (S) 0.20 0.07 12 2.93 0.013 
Mean score of schools (S) 0.15 0.11 = 12 1.29 0.223 
GENDER -0.39 1.32 583 -0.30 0.766 
English Language Learners 2.59 1.76 583 1.47 0.143 
African American 0.63 1.70 583 0.37 0.708 
Grade 8 Year 2 21.65 2.75 583 7.88 0.000 
Grade 7 Year 3 -1.22 3.42 583 -0.36 0.721 
Grade 7 Year 3 2.36 2.62 583 0.90 0.368 
Baseline NJ score 9.54 1.10 583 8.64 0.000 

Random Effects 

Variance Components Estimate ICC 

Level-2 Random Intercept School 5.44 0.012 

Level-1 Residual Student 452.80 
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B10. Analysis Group 3 —- Comprehension -- 3 years of treatment for 8th 
graders 


Table 31. Comprehension — Overall 


Fixed Effects 
Standard 

Effect Estimate Error DF t Value Pr> |t| 
Intercept 640.33 1.16 12 553.36 0.000 
Treatment (S) 1.41 1.61 12 0.88 0.398 
Num eligible students (S) -0.04 0.04 12 -1.22 0.245 
Yr in need of improvement (S) -0.27 0.59 12 -0.45 0.660 
Num. ELL students (S) 0.09 0.09 12 0.96 0.358 
Num. Special Education students (S) 0.01 0.06 12 0.18 0.861 
Mean score of schools (S) -0.05 0.15 12 -0.36 0.727 
GENDER 0.70 1.37 920 0.51 0.608 
English Language Learners 2.07 2.36 920 0.88 0.382 
Special Education student -4.22 1.88 920 -2.24 0.025 
African American -0.94 1.50 920 -0.63 0.529 
Grade 8 Year 4 -0.59 2.37 920 -0.25 0.803 
Grade 8 Year 5 -2.76 1.54 920 -1.79 0.073 
Baseline NJ score 10.27 1.22 920 8.43 0.000 

Random Effects 

Variance Components Estimate ICC 

Level-2 Random Intercept School S77 0.023 

Level-1 Residual Student 421.55 


B-33 


Table 32. Comprehension — Female 


Fixed Effects 
Standard 

Effect Estimate Error DF t Value Pr> |t| 
Intercept 643.80 1.14 12 564.97 0.000 
Treatment (S) -1.44 2.09 12 -0.69 0.504 
Num eligible students (S) -0.04 0.05 12 -0.71 0.494 
Yr in need of improvement (S) -0.85 0.63 12 -1.35 0.201 
Num. ELL students (S) 0.09 0.09 12 1.03 0.323 
Num. Special Education students (S) -0.07 0.08 12 -0.90 0.385 
Mean score of schools (S) 0.08 0.18 12 0.47 0.644 
English Language Learners 2.60 2.89 385 0.90 0.368 
Special Education student -6.32 2.81 385 -2,.25 0.025 
African American 2.13 2.12 385 1.01 0.316 
Grade 8 Year 4 -2.86 3.01 385 -0.95 0.344 
Grade 8 Year 5 -2.19 2.14 385 -1.03 0.306 
Baseline NJ score 10.93 1.92 385 5.68 0.000 

Random Effects 

Variance Components Estimate ICC 

Level-2 Random Intercept School 9.46 0.023 

Level-1 Residual Student 399.32 

Table 33. Comprehension — Male 
Fixed Effects 
Standard 

Effect Estimate Error |= DF t Value Pr> |t| 
Intercept 638.07 1.75 12 = 365.02 0.000 
Treatment (S) 3.19 1.98 12 1.61 0.133 
Num eligible students (S) -0.04 0.05 12 -0.86 0.407 
Yr in need of improvement (S) -0.08 0.77 12 -0.10 0.922 
Num. ELL students (S) 0.01 0.13 12 0.09 0.928 
Num. Special Education students (S) 0.08 0.08 12 1.10 0.295 
Mean score of schools (S) 0.04 0.24 12 0.15 0.881 
English Language Learners 1.03 2.63 523 0.39 0.694 
Special Education student -2.84 1.93 523 -1.47 0.142 
African American -3.11 1.40 523 -2.23 0.026 
Grade 8 Year 4 1.08 3:25 525 0.33 0.739 
Grade 8 Year 5 -2.53 1.98 523 -1.28 0.203 
Baseline NJ score 9.82 0.92 523 10.64 0.000 
Random Effects 

Variance Components Estimate ICC 

Level-2 Random Intercept School 13.48 0.030 

Level-1 Residual Student 434.31 
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Table 34. Comprehension — African American 


Fixed Effects 
Standard 
Effect Estimate Error DF t Value Pr> |t| 
Intercept 638.14 1.82 12 = 350.01 0.000 
Treatment (S) 2.66 2.36 12 1.13 0.281 
Num eligible students (S) -0.09 0.06 12 -1.49 0.162 
Yr in need of improvement (S) 0.39 0.99 12 0.39 0.700 
Num. ELL students (S) 0.20 0.15 12 1.29 0.222 
Num. Special Education students (S) 0.09 0.08 12 Ish5 0.272 
Mean score of schools (S) -0.07 0.20 12 -0.33 0.747 
GENDER -0.99 2.11 496 -0.47 0.640 
Special Education student -7.96 2.09 496 -3.81 0.000 
Grade 8 Year 4 0.37 2:22 496 0.17 0.866 
Grade 8 Year 5 -1.93 1.72 496 -1.12 0.263 
Baseline NJ score 10.51 1.09 496 9.62 0.000 
Random Effects 
Variance Components Estimate ICC 
Level-2 Random Intercept School 15.71 0.033 
Level-1 Residual Student 457.03 
Table 35. Comprehension — Hispanic 
Fixed Effects 
Standard 
Effect Estimate Error DF _ tValue Pr> |t| 
Intercept 643.60 2.49 10. = 258.05 0.000 
Treatment (S) 1.20 2.43 10 0.49 0.632 
Num eligible students (S) -0.02 0.04 10 -0.47 0.650 
Yr in need of improvement (S) -0.58 0.40 10 -1.45 0.177 
Num. ELL students (S) 0.11 0.10 10 1:12 0.290 
Num. Special Education students (S) -0.10 0.08 10 -1.34 0.212 
Mean score of schools (S) -0.22 0.19 10 -1.15 0.277 
GENDER 3.07 1.75 392 1.75 0.081 
English Language Learners 3.76 2.34 392 1.60 0.109 
Special Education student 0.49 2.11 392 0.23 0.817 
Grade 8 Year 4 -2.28 442 392 -0.51 0.607 
Grade 8 Year 5 -5.08 248 392 -2.05 0.041 
Baseline NJ score 10.68 1.85 392 5.76 0.000 
Random Effects 
Variance Components Estimate ICC 
Level-2 Random Intercept School 1.87 0.005 
Level-1 Residual Student 378.39 
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Table 36. Comprehension — Special Education 


Fixed Effects 
Standard 

Effect Estimate Error DF t Value Pr> |t| 
Intercept 635.19 1.82 12 348.72 0.000 
Treatment (S) -0.71 1.67 12 -0.43 0.677 
Num eligible students (S) -0.05 0.04 12 -1.17 0.266 
Yr in need of improvement (S) -0.35 0.50 12 -0.71 0.493 
Num. ELL students (S) 0.07 0.06 12 1.34 0.206 
Num. Special Education students (S) 0.04 0.08 12 0.45 0.658 
Mean score of schools (S) 0.04 0.23 12 0.19 0.849 
GENDER 4.12 1.81 383 2.28 0.023 
English Language Learners 4.31 3.27 383 1.32 0.188 
African American -3,92 1.82 383 -2.15 0.032 
Grade 8 Year 4 -4.12 2.44 383 -1.69 0.091 
Grade 8 Year 5 -3.97 3.10 383 -1.28 0.201 
Baseline NJ score 10.86 1.28 383 8.48 0.000 

Random Effects 

Variance Components Estimate ICC 

Level-2 Random Intercept School 1.36 0.003 

Level-1 Residual Student 442.51 
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B7. Analysis Group 1 — Language Arts -- 1 year of treatment for 6th, 7th, & 
8th graders combined 


Table 37. Language Arts — Overall 


Fixed Effects 
Standard 

Effect Estimate Error DF  t Value Pr> |t| 
Intercept 598.40 0.49 12 = 1218.48 0.000 
Treatment (S) 0.70 0.79 12 0.88 0.397 
Num eligible students (S) -0.10 0.02 12 -5.15 0.000 
Yr in need of improvement (S) 0.41 0.34 12 1.20 0.253 
Num. ELL students (S) 0.04 0.02 12 1.87 0.086 
Num. Special Education students (S) 0.20 0.02 12 9.80 0.000 
Mean score of schools (S) 0.61 0.22 12 2.83 0.016 
GENDER -6.60 1.01 2201 -6.53 0.000 
English Language Learners -4.13 1.32 2201 -3.13 0.002 
Special Education student -11.16 1.06 2201 -10.49 0.000 
African American -2.03 0.90 2201 -2.26 0.024 
Grade 7 Year 1 14.59 2.30 2201 6.35 0.000 
Grade 8 Year 1 27.02 1.44 2201 18.75 0.000 
Grade 6 Year 2 -2.12 1.64 2201 -1.29 0.196 
Grade 6 Year 3 -0.93 1.58 2201 -0.59 0.557 
Grade 6 Year 4 0.93 249 2201 0.37 0.708 
Baseline NJ score 5.68 0.62 2201 9.13 0.000 

Random Effects 

Variance Components Estimate ICC 

Level-2 Random Intercept School 2.46 0.006 

Level-1 Residual Student 440.12 


Table 38. Language Arts — Female 


Fixed Effects 
Standard 

Effect Estimate Error DF t Value Pr> |t| 
Intercept 603.00 0.83 12. 722.37 0.000 
Treatment (S) 2.10 1.32 12 1.59 0.137 
Num eligible students (S) -0.17 0.04 12 -4.16 0.001 
Yr in need of improvement (S) 0.00 0.39 12 -0.01 0.992 
Num. ELL students (S) 0.12 0.05 12 2.55 0.026 
Num. Special Education students (S) 0.26 0.07 12 3.98 0.002 
Mean score of schools (S) 0.48 0.18 12 2.59 0.024 
English Language Learners -5.16 1.84 919 -2.80 0.006 
Special Education student -13.66 1.20 919 -11.40 0.000 
Aftican American -2.19 1.51 919 -1.45 0.147 
Grade 7 Year 1 13.37 2.90 919 4.61 0.000 
Grade 8 Year 1 26.09 2.54 919 10.27 0.000 
Grade 6 Year 2 -0.51 2.39 919 -0.21 0.833 
Grade 6 Year 3 -0.29 2.03 919 -0.14 0.888 
Grade 6 Year 4 0.32 2.82 919 0.11 0.911 
Baseline NJ score 6.91 0.85 919 8.11 0.000 

Random Effects 

Variance Components Estimate ICC 

Level-2 Random Intercept School 0.30 0.001 

Level-1 Residual Student 452.26 
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Table 39. Language Arts — Male 


Fixed Effects 
Standard 

Effect Estimate Error DF tValue Pr> |t| 
Intercept 594.96 0.92 12 644.73 0.000 
Treatment (S) 0.16 1.29 12 0.13 0.902 
Num eligible students (S) -0.06 0.04 12 -1.42 0.181 
Yr in need of improvement (S) 0.57 0.51 12 1.11 0.289 
Num. ELL students (S) -0.03 0.04 12 -0.74 0.475 
Num. Special Education students (S) 0.14 0.06 12 2.20 0.048 
Mean score of schools (S) 0.57 0.25 12 2.27 0.043 
English Language Learners -3.64 1.79 1267 -2.03 0.042 
Special Education student -9.59 1.54 1267 -6.24 0.000 
Aftican American -2.51 1.30 1267 -1.93 0.054 
Grade 7 Year 1 15.92 2.45 1267 6.50 0.000 
Grade 8 Year 1 28.45 1.43 1267 19.97 0.000 
Grade 6 Year 2 -2.91 1.87 1267 -1.55 0.120 
Grade 6 Year 3 -1.17 2.14 1267 -0.55 0.585 
Grade 6 Year 4 1.48 2.86 1267 0.52 0.604 
Baseline NJ score 4.87 0.72 1267 6.80 0.000 

Random Effects 

Variance Components Estimate ICC 

Level-2 Random Intercept School 8.82 0.020 

Level-1 Residual Student 426.64 
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Table 40. Language Arts —African American 


Fixed Effects 
Standard 

Effect Estimate Error | DF t Value Pr> |t| 
Intercept 597.63 0.70 12.00 858.477 0.00 
Treatment (S) 1.73 1.15 12.00 1.499 0.16 
Num eligible students (S) -0.10 0.03 12.00 -3.724 0.00 
Yr in need of improvement (S) 0.35 0.37 12.00 0.943 0.36 
Num. ELL students (S) 0.06 0.04 12.00 1.417 0.18 
Num. Special Education students (S) 0.15 0.03 12.00 6.038 0.00 
Mean score of schools (S) 0.34 0.21 12.00 1.638 0.13 
GENDER -6.80 1.19 = 1226.00 -5.729 0.00 
English Language Learners -8.31 6.48 1226.00 -1.283 0.20 
Special Education student -10.29 1.25 1226.00 -8.228 0.00 
Grade 7 Year 1 14.01 2.19 1226.00 6.409 0.00 
Grade 8 Year 1 25.56 1.97 1226.00 12.993 0.00 
Grade 6 Year 2 -0.07 2.19 1226.00 -0.032 0.98 
Grade 6 Year 3 -0.47 2.13 1226.00 -0.221 0.83 
Grade 6 Year 4 0.47 2.50 1226.00 0.189 0.85 
Baseline NJ score 6.44 0.67 1226.00 9.579 0.00 

Random Effects 

Variance Components Estimate ICC 

Level-2 Random Intercept School 2.00 0.005 

Level-1 Residual Student 432.66 
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Table 41. Language Arts — Hispanic 


Fixed Effects 
Standard 

Effect Estimate Error DF t Value Pr> |t| 
Intercept 599.61 1.01 11 594.78 0.000 
Treatment (S) -0.25 1.17 11 -0,.22 0.833 
Num eligible students (S) -0.13 0.03 11 -4,38 0.001 
Yr in need of improvement (S) 0.37 0.34 11 1.11 0.290 
Num. ELL students (S) -0.01 0.04 11 -0.33 0.745 
Num. Special Education students (S) 0.27 0.05 1-1 5.83 0.000 
Mean score of schools (S) 0.47 0.15 11 3.10 0.011 
GENDER -6.07 1.73) 915 -3.50 0.001 
English Language Learners -3.67 1.21 915 -3.04 0.003 
Special Education student -12.14 1.91 915 -6.34 0.000 
Grade 7 Year 1 15.54 3.25 915 4.78 0.000 
Grade 8 Year 1 28.95 1.94 915 14.91 0.000 
Grade 6 Year 2 -5/12 1.68 915 -3.04 0.003 
Grade 6 Year 3 -1.32 2.20 915 -0.60 0.550 
Grade 6 Year 4 1.44 3.68 915 0.39 0.696 
Baseline NJ score 4.94 0.85 915 5.82 0.000 

Random Effects 

Variance Components Estimate ICC 

Level-2 Random Intercept School 1.64 0.004 

Level-1 Residual Student 459.22 
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Table 42. Language Arts — Special Education 


Fixed Effects 
Standard 

Effect Estimate Error DF t Value Pr> |t| 
Intercept 588.63 0.92 12 636.76 0.000 
Treatment (S) 1.06 1.38 12 0.77 0.459 
Num eligible students (S) -0.03 0.03 12 -0.97 0.351 
Yr in need of improvement (S) 0.71 0.47 12 1.50 0.160 
Num. ELL students (S) 0.05 0.04 12 1.24 0.239 
Num. Special Education students (S) 0.05 0.04 12 1.07 0.307 
Mean score of schools (S) -0.14 0.15 12 -0.92 0.374 
GENDER -3.86 1.44 868 -2.68 0.008 
English Language Learners 1.65 2.14 868 0.77 0.440 
Aftican American -0.61 1.53 868 -0.40 0.693 
Grade 7 Year 1 12.75 3.96 868 3.22 0.002 
Grade 8 Year 1 26.23 1.72 868 15.29 0.000 
Grade 6 Year 2 -6.04 1.76 868 -3,44 0.001 
Grade 6 Year 3 -3.33 2.13 868 -1.57 0.118 
Grade 6 Year 4 -2.25 3.09 868 -0.73 0.468 
Baseline NJ score 4.70 0.64 868 7.34 0.000 

Random Effects 

Variance Components Estimate ICC 

Level-2 Random Intercept School 3.24 0.009 

Level-1 Residual Student 365.01 
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B8. Analysis Group 2 — Comprehension -- 2 years of treatment for 7th & 


8th graders combined 


Table 43. Language Arts — Overall 


Fixed Effects 
Standard 

Effect Estimate Error DF tValue Pr> |t| 
Intercept 609.12 1.41 12 432.73 0.000 
Treatment (S) 2.11 1.93 12 1.10 0.295 
Num eligible students (S) -0.08 0.05 12 -1.42 0.182 
Yr in need of improvement (S) -0.12 0.43 12 -0.29 0.779 
Num. ELL students (S) 0.17 0.07 12 2.33 0.038 
Num. Special Education students (S) 0.07 0.10 12 0.69 0.506 
Mean score of schools (S) 0.15 0.19 12 0.79 0.446 
GENDER -6.09 0.95 1385 -6.39 0.000 
English Language Learners -3.58 2.25 1385 -1.59 0.113 
Special Education student -8.24 1.36 ©1385 -6.07 0.000 
African American -0.63 1.38 1385 -0.46 0.645 
Grade 8 Year 2 13.92 2.53 1385 5:51 0.000 
Grade 7 Year 3 -1.59 2.39 1385 -0.67 0.506 
Grade 7 Year 4 -2.70 2.31 1385 -1.17 0.242 
Baseline NJ score 9.15 0.88 1385 10.41 0.000 

Random Effects 

Variance Components Estimate ICC 

Level-2 Random Intercept School 15.41 0.036 

Level-1 Residual Student 414.95 
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Table 44. Language Arts — Female 


Fixed Effects 
Standard 
Effect Estimate Error DF t Value Pr> |t| 
Intercept 615.12 122°, 12 504.00 0.000 
Treatment (S) 1.48 144 12 1.02 0.327 
Num eligible students (S) -0.09 0.04 12 -2.34 0.038 
Yr in need of improvement (S) 0.19 0.38 12 0.50 0.627 
Num. ELL students (S) 0.21 0.06 12 3.30 0.007 
Num. Special Education students (S) 0.07 0.09 12 0.81 0.436 
Mean score of schools (S) 0.35 0.15 12 2.27 0.043 
English Language Learners -4.04 2.57 596 157 0.117 
Special Education student -10.74 1.73 596 -6.22 0.000 
African American -0.77 2.20 596 -0.35 0.726 
Grade 8 Year 2 9.50 2.81 596 3:39 0.001 
Grade 7 Year 3 -4.14 2.59 596 -1.60 0.110 
Grade 7 Year 4 -5.23 2.38 596 -2.20 0.028 
Baseline NJ score 10.27 1.12 596 9.18 0.000 
Random Effects 
Variance Components Estimate ICC 
Level-2 Random Intercept School 2.89 0.007 
Level-1 Residual Student 439.78 
Table 45. Language Arts — Male 
Fixed Effects 
Standard 
Effect Estimate Error DF t Value Pr> |t| 
Intercept 604.59 1.69 12 358.64 0.000 
Treatment (S) 2.43 2.38 12 1.02 0.328 
Num eligible students (S) -0.06 0.07 12 -0.86 0.406 
Yr in need of improvement (S) -0.38 0.60 12 -0.63 0.542 
Num. ELL students (S) 0.12 0.08 12 1.48 0.166 
Num. Special Education students (S) 0.06 0.12 12 0.52 0.610 
Mean score of schools (S) 0.13 0.23 12 0.55 0.589 
English Language Learners -2.90 2A E76 -1.38 0.169 
Special Education student -6.66 1.77 776 -3.77 0.000 
African American -0.25 1.72 776 -0.15 0.883 
Grade 8 Year 2 17.11 2.69 776 6.37 0.000 
Grade 7 Year 3 0.86 3.09 776 0.28 0.780 
Grade 7 Year 4 -1.01 2.84 776 -0.36 0.721 
Baseline NJ score 8.45 0.91 776 9.28 0.000 
Random Effects 
Variance Components Estimate ICC 
Level-2 Random Intercept School 21.21 0.051 
Level-1 Residual Student 394.11 
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Table 46. Language Arts — African American 


Fixed Effects 
Standard 

Effect Estimate Error DF tValue Pr> |t| 
Intercept 608.82 1.58 12 385.85 0.000 
Treatment (S) 2.26 2.24 12 1.01 0.333 
Num eligible students (S) -0.06 0.06 12 -1.02 0.329 
Yr in need of improvement (S) -0.39 0.59 12 -0.66 0.524 
Num. ELL students (S) 0.20 0.08 12 2.66 0.021 
Num. Special Education students (S) 0.06 0.11 12 0.54 0.600 
Mean score of schools (S) 0.05 0.18 12 0.28 0.784 
GENDER -5.50 1.09 744 -5.03 0.000 
English Language Learners -6.58 3.76 744 -1.75 — 0.080 
Special Education student -8.70 1.89 744 -4.60 0.000 
Grade 8 Year 2 17.23 2.35 744 7.34 0.000 
Grade 7 Year 3 1.72 3.18 744 0.54 0.589 
Grade 7 Year 4 0.45 2.47 744 0.18 0.856 
Baseline NJ score 8.06 1.04 744 7.73 0.000 

Random Effects 

Variance Components Estimate ICC 

Level-2 Random Intercept School 24.77 0.060 

Level-1 Residual Student 385.47 


Table 47. Language Arts — Hispanic 


Fixed Effects 
Standard 

Effect Estimate Error DF tValue Pr> |t| 
Intercept 609.28 1.67 10 364.26 0.000 
Treatment (S) 3.49 1.68 10 2.08 0.064 
Num eligible students (S) -0.20 0.04 10 -5,.23 0.000 
Yr in need of improvement (S) 0.39 0.31 10 1.27 0.234 
Num. ELL students (S) 0.10 0.07 10 1.55 0.151 
Num. Special Education students (S) 0.33 0.07 10 5.03 0.000 
Mean score of schools (S) 0.23 0.14 10 1.69 0.121 
GENDER -7.75 1.67 593 -4.64 0.000 
English Language Learners -2.48 1.61 593 -1.54 0.124 
Special Education student -8.07 1.89 593 -4.27 0.000 
Grade 8 Year 2 9.33 3.03 593 3.08 0.003 
Grade 7 Year 3 -5.18 3.33 593 -1.56 0.120 
Grade 7 Year 4 -6.68 3.74 593 -1.79 0.074 
Baseline NJ score 10.53 1.21 593 8.68 0.000 

Random Effects 

Variance Components Estimate ICC 

Level-2 Random Intercept School 0.09 0.000 

Level-1 Residual Student 448.52 
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Table 48. Language Arts — Special Education 


Fixed Effects 
Standard 

Effect Estimate Error DF t Value Pr> |t| 
Intercept 600.37 2.13 12 281.85 0.000 
Treatment (S) 1.76 2.55 12 0.69 0.504 
Num eligible students (S) -0.11 0.07 12 -1.67 0.120 
Yr in need of improvement (S) 0.47 0.52 12 0.92 0.377 
Num. ELL students (S) 0.27 0.08 12 3.47 0.005 
Num. Special Education students (S) 0.03 0.09 12 0.28 0.783 
Mean score of schools (S) 0.17 0.17 = 12 0.98 0.345 
GENDER -2.83 1.42 575 -2.00 0.046 
English Language Learners -0.25 2.63 575 -0.10 0.924 
African American 0.17 1.76 575 0.10 0.924 
Grade 8 Year 2 16.02 2.90 575 5.53 0.000 
Grade 7 Year 3 -0.71 3.21 575 -0,.22 0.824 
Grade 7 Year 4 -0.80 2.64 575 -0.30 0.762 
Baseline NJ score 8.62 1.12) 575 7.73 0.000 

Random Effects 

Variance Components Estimate ICC 

Level-2 Random Intercept School 21.25 0.046 

Level-1 Residual Student 439.20 
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B9. Analysis Group 3 — Comprehension -- 3 years of treatment for 8th 


graders 


Table 49. Language Arts — Overall 


Fixed Effects 
Standard 

Effect Estimate Error DF tValue Pr> |t| 
Intercept 621.48 0.88 12 705.52 0.000 
Treatment (S) 1.67 1.61 12 1.04 0.320 
Num eligible students (S) -0.06 0.03. 12 -1.82 0.094 
Yr in need of improvement (S) -0.11 0.49 12 -0.23 0.823 
Num. ELL students (S) 0.19 0.07 12 2.48 0.029 
Num. Special Education students (S) -0.08 0.05 12 -1.51 0.156 
Mean score of schools (S) -0.22 0.16 12 -1.34 0.206 
GENDER -5.63 1.23, 917 -4.59 0.000 
English Language Learners -1.58 1.48 917 -1.07 0.285 
Special Education student -6.31 1.42 917 -4.46 0.000 
African American 0.66 Lat 917 0.48 0.631 
Grade 8 Year 4 -3.85 2.04 917 -1.89 0.059 
Grade 8 Year 5 -2.60 1.83. 917 -1.42 0.157 
Baseline NJ score 9.09 1.21 917 7.49 0.000 

Random Effects 

Variance Components Estimate ICC 

Level-2 Random Intercept School 8.81 0.021 

Level-1 Residual Student 412.12 
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Table 50. Language Arts — Female 


Fixed Effects 
Standard 

Effect Estimate Error DF tValue Pr> |t| 
Intercept 626.67 1.23 12 509.50 0.000 
Treatment (S) 0.65 2.35 12 0.28 0.786 
Num eligible students (S) -0.05 0.06 12 -0.76 0.463 
Yr in need of improvement (S) 0.16 0.90 12 0.18 0.862 
Num. ELL students (S) 0.28 0.12 12 2.34 0.037 
Num. Special Education students (S) -0.13 0.08 12 -1.55 0.146 
Mean score of schools (S) -0.22 0.25 12 -0.88 0.397 
English Language Learners -3.06 3.27 383 -0.94 0.349 
Special Education student -8.05 1.76 383 -4.57 0.000 
African American 2.46 2.80 383 0.88 0.381 
Grade 8 Year 4 -4.60 2.90 383 -1.58 0.114 
Grade 8 Year 5 -1.29 2.57 383 -0.50 0.615 
Baseline NJ score 9.74 1.53 383 6.38 0.000 

Random Effects 

Variance Components Estimate ICC 

Level-2 Random Intercept School 20.72 0.046 

Level-1 Residual Student 434.36 

Table 51. Language Arts — Male 
Fixed Effects 
Standard 

Effect Estimate Error DF t Value Pr> |t| 
Intercept 617.57 1.36 12 454.70 0.000 
Treatment (S) 2.31 1.66 12 1.39 0.189 
Num eligible students (S) -0.07 0.03 12 -2.66 0.021 
Yr in need of improvement (S) -0.46 0.63 12 -0.72 0.483 
Num. ELL students (S) 0.09 0.08 12 1.09 0.297 
Num. Special Education students (S) -0.03 0.06 12 -0.57 0.580 
Mean score of schools (S) -0.17 0.22 12 -0.77 0.459 
English Language Learners -0.30 2.11 522 -0.14 0.889 
Special Education student -5.05 2.09 522 -2.42 0.016 
African American -0.12 1.51 522 -0.08 0.939 
Grade 8 Year 4 -2.74 2.40 522 -1.14 0.256 
Grade 8 Year 5 -3,24 1.72 522 -1.88 0.060 
Baseline NJ score 8.68 1.37 522 6.33 0.000 

Random Effects 

Variance Components Estimate ICC 

Level-2 Random Intercept School 9.16 0.023 

Level-1 Residual Student 393.81 
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Table 52. Language Arts — African American 


Fixed Effects 
Standard 

Effect Estimate Error DF t Value Pr> |t| 
Intercept 620.64 1.01 12 613.59 0.000 
Treatment (S) 2.53 1.72 12 1.47 0.167 
Num eligible students (S) -0.02 0.04 12 -0.60 0.558 
Yr in need of improvement (S) -0.75 0.57 12 -1.30 0.217 
Num. ELL students (S) 0.12 0.10 12 1.28 0.226 
Num. Special Education students (S) -0.10 0.05 12 -1.91 0.080 
Mean score of schools (S) -0.17 0.16 12 -1.06 0.311 
GENDER -5.63 1.70 493 -3.31 0.001 
Special Education student -6.98 1.69 493 -4.13 0.000 
Grade 8 Year 4 -2.20 2.01 493 -1.09 0.275 
Grade 8 Year 5 0.55 1.79 493 0.31 0.759 
Baseline NJ score 10.18 1.12 493 9.05 0.000 

Random Effects 

Variance Components Estimate ICC 

Level-2 Random Intercept School 6.25 0.016 

Level-1 Residual Student 388.57 
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Table 53. Language Arts — Hispanic 


Fixed Effects 
Standard 

Effect Estimate Error DF tValue Pr> |t| 
Intercept 625.32 1:39 10 448.99 0.000 
Treatment (S) 0.77 1.70 10 0.45 0.661 
Num eligible students (S) -0.14 0.04 =10 -3.42 0.007 
Yr in need of improvement (S) 0.52 0.40 10 1.33 0.215 
Num. ELL students (S) 0.29 0.08 10 3.74 0.004 
Num. Special Education students (S) -0.04 0.06 10 -0.67 0.519 
Mean score of schools (S) -0.43 0.13 10 -3.16 0.011 
GENDER -6.09 2.09 392 -2.91 0.004 
English Language Learners -1.61 1.50 392 -1.07 0.285 
Special Education student -4.49 2.17 392 -2.08 0.038 
Grade 8 Year 4 -6.15 3.67 392 -1.68 0.094 
Grade 8 Year 5 -6.99 2.43 392 -2.87 0.005 
Baseline NJ score 8.34 2.30 392 3.63 0.001 

Random Effects 

Variance Components Estimate ICC 

Level-2 Random Intercept School 6.74 0.015 

Level-1 Residual Student 429.79 


Table 54. Language Arts — Special Education 


Fixed Effects 
Standard 

Effect Estimate Error DF tValue Pr> |t| 
Intercept 613.22 1.28 12 478.25 0.000 
Treatment (S) 3.09 1.48 12 2.09 0.058 
Num eligible students (S) -0.07 0.04 12 -1.67 0.120 
Yr in need of improvement (S) -0.16 0.50 12 -0.31 0.763 
Num. ELL students (S) 0.16 0.05 12 3.29 0.007 
Num. Special Education students (S) -0.04 0.07 12 -0.57 0.579 
Mean score of schools (S) -0.12 0.20 12 -0.63 0.538 
GENDER -3,22 1.52 381 -2.12 0.034 
English Language Learners -1.49 2.31 381 -0.64 0.521 
African American -0.78 2.35 381 -0.33 0.740 
Grade 8 Year 4 -4,84 1.62 381 -2.99 0.003 
Grade 8 Year 5 -3.06 2.81 381 -1.09 0.278 
Baseline NJ score 8.73 1.20 381 7.25 0.000 

Variance Components Estimate ICC 

Level-2 Random Intercept School 1.53 0.004 

Level-1 Residual Student 353.57 
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B10. Analysis Group 1— Attendance -- 1 year of treatment for 6th, 7th, & 8th 
graders combined 


Table 55. Attendance — Overall 


Fixed Effects 
Standard 

Effect Estimate Error DF t Value Pr > |t| 
Intercept 19.27 0.06 12 46.64 0.000 
Treatment (S) 1.06 0.08 12 0.79 0.443 
Num eligible students (S) 1.00 0.00 12 2.09 0.059 
Yr in need of improvement (S) 1.04 0.03 12 1.32 0.212 
Num. ELL students (S) 1.00 0.00 12 0.24 0.817 
Num. Special Education students (S) 0.99 0.00 12 -2.37 0.036 
Mean score of schools (S) 0.98 0.02 12 -0.98 0.345 
GENDER 0.98 0.04 2282 -0.47 0.636 
English Language Learners 0.70 0.05 2282 -7.17 0.000 
Special Education student 0.89 0.05 2282 -2,36 0.018 
Aftican American 0.98 0.05 2282 -0.46 0.647 
Grade 7 Year 1 1.17 0.06 2282 2.59 0.010 
Grade 8 Year 1 1.42 0.07 2282 4.92 0.000 
Grade 6 Year 2 1.14 0.08 2282 1.76 0.078 
Grade 6 Year 3 0.69 0.08 2282 -4.42 0.000 
Grade 6 Year 4 0.45 0.13 2282 -6.24 0.000 
Baseline NJ score 0.95 0.02 2282 -2.65 0.009 


Table 56. Attendance — Female 


Fixed Effects 
Standard 
Effect Estimate Error DF t Value Pr> |t| 
Intercept 18.50 0.06 12 52.19 0.000 
Treatment (S) 1.10 0.10 12 0.98 0.348 
Num eligible students (S) 1.01 0.00 12 3.05 0.011 
Yr in need of improvement (S) 1.02 0.04 12 0.56 0.584 
Num. ELL students (S) 1.00 0.00 12 -0.56 0.587 
Num. Special Education students (S) 0.99 0.01 12 -2.36 0.036 
Mean score of schools (S) 0.99 0.02 12 -0.68 0.508 
English Language Learners 0.71 0.06 965 -6.12 0.000 
Special Education student 0.88 0.09 965 -1.48 0.140 
Aftican American 1.05 0.07 965 0.67 0.505 
Grade 7 Year 1 1.19 0.07 965 2.58 0.010 
Grade 8 Year 1 1.62 0.08 965 5.78 0.000 
Grade 6 Year 2 1.26 0.14 965 1.71 0.087 
Grade 6 Year 3 0.73 0.09 965 -3.59 0.001 
Grade 6 Year 4 0.48 0.10 965 -7.42 0.000 
Baseline NJ score 0.97 0.03 965 -0.88 0.378 
Table 57. Attendance — Male 
Fixed Effects 
Standard 
Effect Estimate Error DF t Value Pr> |t| 
Intercept 19.90 0.07 12 41.55 0.000 
Treatment (S) 1.02 0.07 12 0.30 0.770 
Num eligible students (S) 1.00 0.00 12 0.53 0.607 
Yr in need of improvement (S) 1.05 0.03 12 2.06 0.062 
Num. ELL students (S) 1.00 0.00 12 1.02 0.329 
Num. Special Education students (S) 0.99 0.00 12 -1.54 0.150 
Mean score of schools (S) 0.99 0.01 12 -0.96 0.356 
English Language Learners 0.70 0.07 1302 -5.00 0.000 
Special Education student 0.90 0.07 1302 -1.50 0.134 
African American 0.94 0.06 1302 -1.02 0.307 
Grade 7 Year 1 1.15 0.09 1302 1.61 0.107 
Grade 8 Year 1 1.29 0.08 1302 3.28 0.001 
Grade 6 Year 2 1.04 0.06 1302 0.68 0.498 
Grade 6 Year 3 0.67 0.10 1302 -3.94 0.000 
Grade 6 Year 4 0.44 0.16 1302 -5,22 0.000 
Baseline NJ score 0.94 0.02 1302 -2.74 0.007 
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Table 58. Attendance - African American 


Fixed Effects 
Standard 
Effect Estimate Error DF t Value Pr> |t| 
Intercept 19.40 0.07 12 43.30 0.000 
Treatment (S) 1.11 0.07 12 1.49 0.162 
Num eligible students (S) 1.00 0.00 12 1.08 0.302 
Yr in need of improvement (S) 1.04 0.03. 12 1.54 0.149 
Num. ELL students (S) 1.00 0.00 12 1.16 0.267 
Num. Special Education students (S) 0.99 0.00 12 -1.65 0.124 
Mean score of schools (S) 0.97 0.02 12 -1.85 0.088 
GENDER 0.97 0.06 1317 -0.56 0.578 
English Language Learners 0.37 0.40 1317 -2.46 0.014 
Special Education student 0.87 0.05 1317 -2.74 0.007 
Grade 7 Year 1 1.03 0.10 1317 0.33 0.741 
Grade 8 Year 1 1.37 0.11 1317 2.91 0.004 
Grade 6 Year 2 1.06 0.10 1317 0.58 0.559 
Grade 6 Year 3 0.61 0.11 1317 -4.41 0.000 
Grade 6 Year 4 0.43 0.16 1317 -5.16 0.000 
Baseline NJ score 0.94 0.02 1317 -2.87 0.005 
Table 59. Attendance — Hispanic 
Fixed Effects 
Standard 

Effect Estimate Error DF t Value Pr > |t| 
Intercept 15.76 0.10 12 27.76 0.000 
Treatment (S) 1.33 0.12 12 2.30 0.040 
Num eligible students (S) 1.00 0.00 12 0.48 0.639 
Yr in need of improvement (S) 1.01 0.03 12 0.22 0.830 
Num. ELL students (S) 1.00 0.00 12 0.97 0.351 
Num. Special Education students (S) 1.00 0.00 12 -0.93 0.372 
Mean score of schools (S) 1.00 0.01 12 -0.29 0.779 
GENDER 0.98 0.03 909 -0.49 0.626 
English Language Learners 0.73 0.06 909 -519 0.000 
Special Education student 0.92 0.09 909 -0.84 0.402 
Grade 7 Year 1 1.35 0.05 909 6.16 0.000 
Grade 8 Year 1 1.47 0.09 909 4.47 0.000 
Grade 6 Year 2 1.24 0.07 909 2.92 0.004 
Grade 6 Year 3 0.83 0.13 909 -1.48 0.140 
Grade 6 Year 4 0.48 0.12 909 -6.12 0.000 
Baseline NJ score 0.97 0.03 909 -1.06 0.289 
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Table 60. Attendance - Special Education 


Fixed Effects 
Standard 

Effect Estimate Error DF t Value Pr> |t| 
Intercept 19.48 0.06 12 52.46 0.000 
Treatment (S) 1.04 0.10 12 0.42 0.685 
Num eligible students (S) 1.00 0.00 12 0.67 0.515 
Yr in need of improvement (S) 1.05 0.03 12 1.58 0.139 
Num. ELL students (S) 1.00 0.00 12 0.61 0.553 
Num. Special Education students (S) 0.99 0.01 12 -1.18 0.262 
Mean score of schools (S) 1.00 0.01 12 -0.43 0.677 
GENDER 0.98 0.05 896 -0.36 0.720 
English Language Learners 1.02 0.12 896 0.17 0.869 
African American 0.98 0.07 896 -0.26 0.797 
Grade 7 Year 1 0.98 0.10 896 -0.19 0.847 
Grade 8 Year 1 1.22 0.07 896 2.80 0.006 
Grade 6 Year 2 1.01 0.08 896 0.10 0.919 
Grade 6 Year 3 0.69 0.12 896 -3.16 0.002 
Grade 6 Year 4 0.31 0.21 896 -5.47 0.000 
Baseline NJ score 0.94 0.02 896 -3.12 0.002 
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B11. Analysis Group 2 — Comprehension -- 2 years of treatment for 7th & 8th 


graders combined 


Table 61. Attendance — Overall 


Fixed Effects 
Standard 

Effect Estimate Error DF t Value Pr > |t| 
Intercept 19.09 0.05 12 65.13 0.000 
Treatment (S) 1.05 0.07 12 0.67 0.515 
Num eligible students (S) 1.00 0.00 12 2.31 0.040 
Yr in need of improvement (S) 1.03 0.02 12 1.09 0.299 
Num. ELL students (S) 1.00 0.00 12 -0.61 0.552 
Num. Special Education students (S) 1.00 0.00 12 -1.14 0.277 
Mean score of schools (S) 0.99 0.01 12 -1.07 0.306 
GENDER 1.03 0.06 1088 0.44 0.663 
English Language Learners 0.88 0.07 1088 -1.85 0.065 
Special Education student 0.98 0.05 1088 -0.36 0.718 
African American 1.00 0.10 1088 -0.03 0.974 
Grade 8 Year 2 1.17 0.07 1088 2.34 0.020 
Grade 7 Year 3 0.65 0.09 1088 -4,54 0.000 
Grade 7 Year 4 0.51 0.13 1088 -5.18 0.000 
Baseline NJ score 0.91 0.04 1088 -2.26 0.024 


Table 62. Attendance — Female 


Fixed Effects 
Standard 

Effect Estimate Error DF  t Value Pr> |t| 
Intercept 17.12 0.06 12 48.25 0.000 
Treatment (S) 1.24 0.07 12 3.12 0.010 
Num eligible students (S) 1.01 0.00 12 5.06 0.000 
Yr in need of improvement (S) 0.99 0.02 12 -0.65 0.525 
Num. ELL students (S) 1.00 0.00 12 -1.40 0.187 
Num. Special Education students (S) 1.00 0.00 12 -1.10 0.292 
Mean score of schools (S) 0.98 0.00 12 -4.60 0.001 
English Language Learners 0.87 0.09 485 -1.60 0.110 
Special Education student 0.99 0.09 485 -0.11 0.915 
African American 0.97 0.13 485 -0.20 0.843 
Grade 8 Year 2 1.21 0.09 485 2.09 0.037 
Grade 7 Year 3 0.62 0.10 485 -4.69 0.000 
Grade 7 Year 4 0.58 0.21 485 -2.54 0.012 
Baseline NJ score 0.96 0.03 485 -1.15 0.252 
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Table 63. Attendance - Male 
Fixed Effects 
Standard 
Effect Estimate Error DF _ t Value Pr> |t| 
Intercept 20.37 0.04 12 71.60 0.000 
Treatment (S) 0.93 0.09 12 -0.78 0.452 
Num eligible students (S) 1.00 0.00 12 1.06 0.312 
Yr in need of improvement (S) 1.07 0.03 12 2.36 0.036 
Num. ELL students (S) 1.00 0.00 12 -0.29 0.780 
Num. Special Education students (S) 1.00 0.00 12 -0.73 0.481 
Mean score of schools (S) 1.01 0.01 12 0.82 0.427 
English Language Learners 0.96 0.13 590 -0.34 0.733 
Special Education student 0.97 0.09 590 -0.36 0.716 
African American 1.03 0.10 590 0.30 0.763 
Grade 8 Year 2 1.12 0.11 590 1.07 0.286 
Grade 7 Year 3 0.70 0.13 590 -2.72 0.007 
Grade 7 Year 4 0.48 0.14 590 -5.46 0.000 
Baseline NJ score 0.87 0.06 590 -2.32 0.021 
Table 64. Attendance — African American 
Fixed Effects 
Standard 

Effect Estimate Error DF _ t Value Pr> |t| 
Intercept 20.63 0.07 12 43,35 0.000 
Treatment (S) 0.96 0.10 12 -0.38 0.713 
Num eligible students (S) 1.01 0.00 12 3.09 0.010 
Yr in need of improvement (S) 0.99 0.04 12 -0.20 0.843 
Num. ELL students (S) 0.99 0.01 12 -0.98 0.346 
Num. Special Education students (S) 0.99 0.00 12 -2.43 0.032 
Mean score of schools (S) 0.99 0.01 12 -0.64 0.534 
GENDER 1.06 0.08 613 0.70 0.481 
English Language Learners 0.63 0.15 613 -3.05 0.003 
Special Education student 1.01 0.08 613 0.15 0.879 
Grade 8 Year 2 1.02 0.10 613 0.24 0.811 
Grade 7 Year 3 0.60 0.12 613 -4,22 0.000 
Grade 7 Year 4 0.57 0.14 613 -4.08 0.000 
Baseline NJ score 0.89 0.04 613 -3.23 0.002 
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Table 65. Attendance — Hispanic 


Fixed Effects 
Standard 
Effect Estimate Error DF t Value Pr> |t| 
Intercept 18.52 0.08 9 36.49 0.000 
Treatment (S) 1.18 0.12 9 1.41 0.192 
Num eligible students (S) 1.00 0.00 9 -0.74 0.479 
Yr in need of improvement (S) 1.04 0.03 9 1.35 0.209 
Num. ELL students (S) 1.00 0.00 9 -0.47 0.651 
Num. Special Education students (S) 1.01 0.00 9 2.40 0.040 
Mean score of schools (S) 0.99 0.01 9 -2.28 0.048 
GENDER 0.95 0.06 437 -0.87 0.383 
English Language Learners 0.93 0.08 437 -0.99 0.322 
Special Education student 0.98 0.04 437 -0.46 0.647 
Grade 8 Year 2 1.37 0.07 437 4.19 0.000 
Grade 7 Year 3 0.67 0.11 437 -3.59 0.001 
Grade 7 Year 4 0.44 0.18 437 -4.72 0.000 
Baseline NJ score 0.96 0.06 437 -0.59 0.558 
Table 66. Attendance - Special Education 
Fixed Effects 
Standard 

Effect Estimate Error DF _ t Value Pr> |t| 
Intercept 20.00 0.07 12 43.98 0.000 
Treatment (S) 1.05 0.09 12 0.53 0.607 
Num eligible students (S) 1.01 0.00 12 4.05 0.002 
Yr in need of improvement (S) 1.02 0.03 12 0.63 0.543 
Num. ELL students (S) 0.99 0.00 12 -1.40 0.188 
Num. Special Education students (S) 0.99 0.00 12 -3,.25 0.008 
Mean score of schools (S) 1.00 0.01 12 -0.84 0.419 
GENDER 1.05 0.11 445 0.44 0.663 
English Language Learners 1.07 0.14 445 0.47 0.639 
African American 1.04 0.14 445 0.29 0.769 
Grade 8 Year 2 1.19 0.14 445 1.21 0.227 
Grade 7 Year 3 0.69 0.10 445 -3.60 0.001 
Grade 7 Year 4 0.47 0.13 445 -5.72 0.000 
Baseline NJ score 0.92 0.07 445 -1.11 0.268 


B12. Analysis Group 3 —- Comprehension -- 3 years of treatment for 8th graders 


Table 67. Attendance — Overall 


Fixed Effects 
Standard 

Effect Estimate Error DF t Value Pr> |t| 
Intercept 15.83 0.07 12 37.16 0.000 
Treatment (S) 0.83 0.10 12 -1.91 0.080 
Num eligible students (S) 1.01 0.00 12 3.18 0.009 
Yr in need of improvement (S) 1.00 0.02 12 -0.04 0.968 
Num. ELL students (S) 0.99 0.00 12 -2.98 0.012 
Num. Special Education students (S) 0.99 0.00 12 -2.94 0.013 
Mean score of schools (S) 1.01 0.01 12 1.10 0.293 
GENDER 1.01 0.05 706 0.31 0.756 
English Language Learners 0.79 0.17 706 -1.39 0.166 
Special Education student 1.16 0.09 706 1.69 0.090 
African American 1.01 0.08 706 0.17 0.866 
Grade 8 Year 4 0.91 0.11 706 -0.90 0.368 
Grade 8 Year 5 0.91 0.09 706 -1.12 0.264 
Baseline NJ score 0.91 0.04 706 -2.29 0.022 

Table 68. Attendance — Female 
Fixed Effects 
Standard 

Effect Estimate Error DF t Value Pr> |t| 
Intercept 13.84 0.07 12 39.99 0.000 
Treatment (S) 0.96 0.09 12 -0.43 0.675 
Num eligible students (S) 1.01 0.00 12 3.51 0.005 
Yr in need of improvement (S) 1.00 0.02 12 -0.18 0.864 
Num. ELL students (S) 0.99 0.00 12 -3.00 0.012 
Num. Special Education students (S) 0.99 0.00 12 -2.99 0.012 
Mean score of schools (S) 1.01 0.00 12 2.61 0.023 
English Language Learners 0.97 0.19 306 -0.19 0.853 
Special Education student 1.20 0.12 306 1.47 0.142 
African American 1.10 0.14 306 0.67 0.505 
Grade 8 Year 4 0.89 0.18 306 -0.67 0.505 
Grade 8 Year 5 0.86 0.10 306 -1.48 0.141 
Baseline NJ score 0.93 0.05 306 -1.46 0.145 
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Table 69. Attendance — Male 


Fixed Effects 
Standard 
Effect Estimate Error DF t Value Pr> |t| 
Intercept 17.34 0.10 12 29.53 0.000 
Treatment (S) 0.73 0.13. 12 -2.35 0.037 
Num eligible students (S) 1.00 0.00 12 1.59 0.138 
Yr in need of improvement (S) 1.01 0.03 12 0.16 0.876 
Num. ELL students (S) 1.00 0.00 12 -1.12 0.287 
Num. Special Education students (S) 0.99 0.00 12 -2.19 0.049 
Mean score of schools (S) 1.00 0.01 12 -0.07 0.950 
English Language Learners 0.67 0.18 388 -2.27 0.023 
Special Education student 1.11 0.09 388 1.23 0.221 
African American 0.97 0.10 388 -0.34 0.732 
Grade 8 Year 4 0.93 0.10 388 -0.67 0.501 
Grade 8 Year 5 0.95 0.12 388 -0.42 0.674 
Baseline NJ score 0.90 0.05 388 -2.25 0.025 
Table 70. Attendance — African American 
Fixed Effects 
Standard 
Effect Estimate Error DF t Value Pr> |t| 
Intercept 17.93 0.11 12 25.91 0.000 
Treatment (S) 0.78 0.15 12 -1.70 0.114 
Num eligible students (S) 1.01 0.00 12 3.73 0.003 
Yr in need of improvement (S) 0.91 0.06 12 -1.64 0.126 
Num. ELL students (S) 0.98 0.01 12 -2.90 0.014 
Num. Special Education students (S) 0.99 0.00 12 -2.91 0.014 
Mean score of schools (S) 1.02 0.01 12 1:31 0.215 
GENDER 1.00 0.07 362 0.05 0.964 
Special Education student 1.34 0.08 362 3.56 0.001 
Grade 8 Year 4 1.21 0.13 362 1.40 0.162 
Grade 8 Year 5 0.91 0.11 362 -0.85 0.398 
Baseline NJ score 0.89 0.06 362 -1.94 0.053 
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Table 71. Attendance — Hispanic 


Fixed Effects 
Standard 

Effect Estimate Error DF t Value Pr> |t| 
Intercept 15.44 0.10 10 26:53 0.000 
Treatment (S) 0.89 0.15 10 -0.73 0.481 
Num eligible students (S) 1.00 0.00 10 0.90 0.387 
Yr in need of improvement (S) 1.00 0.06 10 -0.06 0.952 
Num. ELL students (S) 0.98 0.01 10 -1.75 0.110 
Num. Special Education students (S) 1.00 0.01 10 0.49 0.637 
Mean score of schools (S) 1.00 0.01 10 -0.66 0.526 
GENDER 1.06 0.06 317 0.87 0.387 
English Language Learners 0.76 0.15 317 -1.86 0.064 
Special Education student 1.01 0.12 317 0.09 0.926 
Grade 8 Year 4 0.52 0.11 317 -5.80 0.000 
Grade 8 Year 5 0.89 0.08 317 -1.40 0.163 
Baseline NJ score 0.95 0.04 317 -1.35 0.178 


Table 72. Attendance — Special Education 


Fixed Effects 


Effect 


Intercept 

Treatment (S) 

Num eligible students (S) 

Yr in need of improvement (S) 
Num. ELL students (S) 

Num. Special Education students (S) 
Mean score of schools (S) 
GENDER 

English Language Learners 
African American 

Grade 8 Year 4 

Grade 8 Year 5 

Baseline NJ score 


Standard 
Estimate Error 

21.39 0.09 
0.63 0.11 
1.01 0.00 
1.04 0.04 
1.00 0.01 
0.98 0.01 
1.00 0.01 
0.94 0.06 
1.00 0.24 
1.21 0.16 
1.01 0.17 
0.87 0.14 
0.86 0.06 
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DF tValue Pr> |t 


34.65 
-4.34 
2.47 
1.15 
-0.65 
-2.80 
-0.31 
-0.95 
0.01 
1.20 
0.08 
-1.04 
-2.62 


0.000 
0.001 
0.030 
0.274 
0.526 
0.016 
0.761 
0.343 
0.989 
0.231 
0.939 
0.298 
0.010 


Appendix C 


Standard Deviations 


Standard Deviations Used in Calculating Effect Sizes 


Table 73. Summary of Control Group Standard Deviations 


Analysis 


groups 


Outcomes 


Attendance 


Overall 


20.8 


Female 


22.34 


Male 


African 
American 


Special 
Education 


Vocabulary 


31.65 


30.53 


29.59 


30.16 


32.17 


Comprehension 


27.98 


28.19 


Language Arts 


Attendance 


22.44 


21.19 


Vocabulary 


27.16 


26.04 


28.09 


26.86 


26.78 


Comprehension 


26.24 


26.95 


Language Arts 
Attendance 


Vocabulary 


14.11 
28.64 


13.07 
25.07 


Comprehension 


24.88 


23.54 


Language Arts 
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Appendix D 


Measures 
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D3. Striving Readers Observation Tool 


WESTAT 


AN EMPLOYEE-OWNED 
RESEARCH CORPORATION 


Striving Readers: Newark 


CLASSROOM OBSERVATION PROTOCOL 


Spring 2007 


Observer name Date of 
observation 


School: Obs. Start Time End 
Time: 


Teacher name: Teacher gender: Female Male 


Grade you are observing 6"" 7" 8" combination 


Adult present in the room besides the classroom teacher? Yes No 


Role of this adult (e.g., student 
teacher/paraprofessional) 


# students in class 10 minutes into the observation: [# females: # 
males ] 
(please check to ensure that #females + #males sum to the total number of students) 
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I. Physical Environment 


1. Resources (e.g, print materials, technology) 


1 2 3 
Sparsely equipped 


4 


Rich in resources 


2. Bulletin Boards and/or Walls (¢.g, student samples and word walls) 


1 2 3 


Bare, or used solely 
jor decorative purposes 


3. Availability of Books 


1 2 3 


Few books available, 
and/ or one reading level only 


4 


Rich with student work 


and content-relevant materials 


4 


Books plentiful, available, 


and for variety of reading levels 


II. Materials /Technologies Used During Class Period by Students (Please check all that apply. 


1. Reading or discussion of print materials (if yes, complete 1a-1¢) Yes O No O 
la. Novels/Stories/Poems Yes O No O 
1b. Textbook/ Anthology Yes O No O 
1c. Articles Yes O No O 
2. Did students read text during this class period? (i yes, complete 2a) Yes O No O 
2a. Are all students reading the same text? Yes O No O 
3. Workbooks / worksheets Yes O No O 
4. Video, film, tv Yes O No O 
5. Writing in notebooks/journals Yes O No O 
6. Computer use (if yes, complete 6a-6c) Yes O No O 
6a. Used for research (such as web searches) Yes O No O 
6b. Used for writing (MS Word) Yes O No O 
6c. Used for reading instruction (specialty software) Yes O No O 
7. Audio (tape players, cd players; NOT teacher reading aloud) Yes O No O 
III. | Classroom Climate 
To what extent do you agree or disagree with each of the following statements? 
Strongly Disagree Neither agree Agree Strongly 
disagree nor disagree agree 
1. Instructional time was well structured; transitions were well 
1 2 3 4 5 
defined 
2. Participation of all students was actively encouraged by the 
teacher 
3. This appeared to be a safe environment for struggling readers to 
learn in. 
4. At the end of the class period, teacher summarized what was 1 > 3 4 5 
learned 
5. There was a high level of critical thinking required by students 1 2 3 4 5 
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IV. _Time Sampled Data (Begin 1° row ten minutes after the official start time of class and then complete every 10 minutes) 


Ld 


% of student s Student Grouping Instructional Codes 
Time engaged in (refers to how students are working, not how (Add a code of “T” if teacher is providing direct instruction or modeling; Add “S” if students are applying strategies 
task seating is arranged) on their own or with one another; Use both “T” and “S” if applicable) 
<25% oO ae 4 on g Vocabulary Fluency Comprehension Writing Other Activity 
25-50% UO 
1: 
51-75% O | small group OO Individual O 
<75% CO 
<3% O Vocabulary Fluency Comprehension fu 7 
2 55 50% oO Whole class O Pairs O Locabulay atte ee ee anna 
a 51-75% | small group OO Individual O 
<75% LO 
<25% O ny Eluency D ti ther Activi 
3 35 50% oO Whole class O Pairs O Votabula a ee nae erecae 
—— 51-75% | Small group OO Individual O 
<75% C 
<25% 0 ny Fluency D ti ther Activi 
4 95.50% O Whole class Oo Pairs O Vocabular sa cenpnienaee ae oe 
nn 51-75% O | Smallgroup O Individual O 
<75% C 
<25% L] ny Eluency prehensi ti ther Activi 
5 05-50% (] | Wholeclass OO Pairs 0 —— nal a re dail 
; 51-75% O | Smallgroup O Individual O 
<75% CO 
<3% O Vocabulary Fluency Comprehension a eee 
6 95.50% =| Whole class im Pairs O Vosabular ne nee ac mene 
—— 51-75% U1 | Smallgroup OO Individual O 
<75% C 
<5% Vocabulary Fluency Comprehension “ ther AAcivt 
7 25-50% (—] | Wholeclass OO Pairs 0 a oe te ce —— 
; 51-75% OU Small group O Individual O 
<75% CO 
<2% O Vocabulary Fi . se 
8 25-50% Ct =| Whole class O Pairs O Vocabular on aaa sae aan 
<——- 51-75% O | Small group OO Individual O 
<75% CO 
<% O Vocabulary Fluency Comprehension - nee 
9 95.50% =| Whole class im Pairs O Vocabular se oe a Sere 
—=———-7 51-75% UO | smallgroup O Individual O 
<75% O 
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Instruction codes 


Vocabulary 


Focus is on vocabulary development. 


Fluency 


Focus is on improving reading 


Comprehension 


Focus is on improving student reading 


Watung 


Students writing, or writing 


Other Activity 


Describes other activities 


fluency comprehension. instruction is happening. 
Code Definition Code| Definition  |Code Definition Code Definition Code Definition 
Context Clue: Teacher directs : : 5 Collaborative Writing: 
igen TAT ais Choral Reading: Groups : : : 
students to look in rest of text to : Graphic Organizer: Teacher or . teachers and/or students a . 
; ' . of students read in . : : T-CW Administration: Teacher engaged 
.{ infer meaning of a word. Can ok : T-GO] students complete a pictorial work together to create ; er F 
T-Cd . ‘ S-CR unison. Passage may be 4 . ? ‘ : ADM | in administrative tasks, such as 
inc. reading rest of the sentence : . “ S-GO}] representation of how ideas in text é stories, W . 
. read multiple times. : S-CW grading papers 
and guessing what makes sense, are connected and organized. teacher/student as the 
f : Teacher may tead along. : 
or look at accompanying picture. scribe 
Dictionary use: Students look up Guided Writing: Wesessmmene Teacher and 
unfamiliar words. Includes F P : K-W-L: “What I know, what I want Teachers guide writing ASS)||"Saqe- : ; : 
f Paired Reading: Pairs of ‘ 3 poe students engaged in testing 
7 glossary provided by teacher, S-| to find out, what I learned.” 3- sy} Process through mini- 
S-DIC ; : S-PR) students take turns rae ; 2 5 T-GW 
glossary in their textbook, a teadincoacloud KWL] column chart. Fill out K and W before lessons & conferences. C Cc f 
ta: . eal out loud. : : : : ; oH > use 
separate dictionary, or online 8 reading and L after reading. Sometimes called writer’s | S-COM Computer: rome uter Use 10k 
ee i : research, writing or instruction 
dictionary. workshop 
one 4 Cooperative: Students are 
. Monitoring Understanding: Teacher SETI CL 
Repeated oral Reading: ; F : ; working collaboratively in groups 
, 5 monitors by asking specific questions 3 ; . : 
Etymology: Teacher discusses Same passage read & dacotnising Sad cnteioinoniioE ournal Use: Students to discuss text. May inc. pair 
the history or origin of rd. aloud multiple times (by > ; tite in journals/blogs. A . reading, reciprocal teaching, or 
Too ee ne SRR Fore AER HES ” T-MU] own understanding. May do this S-JU aaa es =| 8 S:C@P | eRe e Tee DIOS ae OR: 
Can involve identifying prefixes teacher and/or : journal is usually in a other structured protocols 
: , through a think aloud (add TA code). 2 : : ‘ 
and suffixes. students) while others . ; ; separate notebook around reading or discussing 
Does not inc. general questions, like es 
follow along. ee ‘ - text. Peer or group editing add 
are there ate any questions? patie 
W” to code/ 
: , Making Connections: Teacher or ; ‘ , ‘ , 
< ; , : , Listening to Text: SS Note Taking: Students Discussion: Teacher is leading or 
Graphic Organizer: Pictorial 5 : T-MC}] students relate text to current events F , j ; 
. ; ‘ , Students read along in a ; . are taking notes. If they moderating a class discussion. 
T-GO_ representation of how ideas in a S-LT| sie ae : or to material already covered. T may S-NT| s i T-DIS : 
: book while listening to so topes : : : are copying notes add — There is student to student 
text are connected & organized. S-MC} do this through a think aloud (add a z : ; ; 
the text. X to code interaction. Otherwise use LEC 
TA code). 
: ‘ ( Writing Process: Students 
Pre-teaching: Discuss meaning of bt as ae Lecture: Teacher talks most of 
words before read text. Can Biedichotis Studeuis:azke S wank 80 planting, the time. Students respond 
T-P] . : : ee O-F | Other Fluency: Specify S-P| predictions before, and at specified ep| Writing, revising or T-LEC ; é ae. 
involve discussing word & , : : -WP| hs : briefly to questions. Almost no 
ee ‘ points during reading. editing their text. Long ‘ 
activating prior knowledge. ; student to student talk 
: term project 
; . Modeling: Teacher demonstrates 
Word Wall: List of words related Question Response: 
: T-MOD |} / models how to analyze a word, 
to unit posted on the wall & Students respond to iiewer a cuestion ‘ 
T-WW) easily visible. Use of the word Summarizing: Affer reading students questions or prompts in _ sles 
wall would inc adding new S-SM] use one of a number of strategies to S-QR| writing — could be Think Aloud : Teacher describes 
S-WW)| words, using words on wall to create a summaty. questions at the end of a their thought process to model 
complete a task, or overtly text, from teacher, or on T-TA] how a strategy is used. Literally 
referring to posted words workbooks. walks students through their 
personal thought process. 
Text Structure: Explicitly teaching 
expository text structure. May in \ of Transition: No instruction is 
ee ase id Baie a Guick Waiting: Meant te tain, ee Betse siden are 
O-V | Other Vocabulary: Specify T-TX Serre i S-QW) elicit connection or TRAN BP : ae 
: bold, & recognize signal words (eg : transitioning from one activity to 
“ » . response to reading. ; 
therefore”). T’ may do this through a another 
think aloud (add TA code). 
O-C | Other Comprehension: Specify O-W | Other Writing: Specify OTH | Other: Specify 


VI. Student Questions 


L, Ask a students if this was a typical class (if no, also ask for an example of how it was 
atypical). Record response here: 


Student: 


Student gender: Male | Female 
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D4. Westat Fidelity Measure 


WES TAT 


AN EMPLOYEE-OWNED 
RESEARCH CORPORATION 


Striving Readers: Newark 


FIDELITY PROTOCOL 


Spring 2007 


Observer name Date of observation 


School: Lesson Start Time End Time: 


Teacher name: Teacher gender: Female Male 


Grade you are observing 6" 7" 8"" Mixed 


# students in class 10 minutes into the observation: [# girls: # boys ] 
# of students tardy: 


For how long did this READ 180 section meet today? O < hour O 60-89 min O 90-95 min O 96 min+ 


I. Classroom Organization, Materials, and Equipment 


Yes | No | NA 

1. Room had a space designated for independent reading QO | O 

1a. Independent reading are has comfortable seating QO};O}; 90 

1b. Independent reading area has sufficient working cd players | O | O | O 

1c. Independent reading area has adequate paperback books QO};O}; 90 
2. Room had a space designated for small group instruction OQ | O 
3. Room had a space designated for whole group instruction. OQ | O 
4. Room had a space containing computer workstations. OQ | O 

4a. There are at least five functioning computer workstations QO};O}9O0 
or ee i enough space for students to move easily between ololoa 
5. Room has a paperback library with books labeled by level QO} 9O 
6. Expectations for student performance & behavior are posted QO | O 


II. Instruction 


Whole-Group Instruction Start time: End time 


1. Do the instructional activities involve a READ 180 rBook? 
[_] Yes [_] No 

1b. What color is the cover of the rBook? 
[ ]Blue [_] Green 

2. Do all students have an rBook? 
[_] Yes [_] Some of them have rBooks [_] No 


3. Are students using their rBooks for writing responses to the teacher’s questions and 
prompts? 


[_] Yes L_] Some of them are using rBooks [_] No 
4. Do the students work with any materials other than READ 180? 
[_] Yes [_] No 
v 


If yes, briefly describe the materials below: (remember to ask teacher question 4 at the end of 


class) 


5. Does the teacher attempt to engage all students in the instructional activities by asking 
questions, providing prompts, and soliciting responses? 


[_] Yes L_]Teacher attempts to engage some students [_] No 
6. Does the teacher make explicit connections between the Whole-Group learning activities 


and the content or focus of the Small-Group instruction that will follow the Whole 
Group session? 


[_] Yes [_] No 


Small-Group Instruction Start time: End time 


1. Do the instructional activities involve a READ 180 rBook? 
[_] Yes [_] No 
2. Do all students have an rBook? 


[_] Yes [_] Some of them have rBooks [_] No 


3. Are students using their rBooks for writing responses to the teacher’s questions and 
prompts? 


[_] Yes |_] Some of them are using rBooks [_] No 
4. Do the students work with any materials other than READ 180? 


[_] Yes [_] No 


If yes, briefly describe the materials below: (remember to ask teacher question 4 at the end of 
class) 


5. Does the teacher attempt to engage all of the students in the small-group instructional 
activities by asking questions, providing prompts, and soliciting responses? 


[_] Yes [_]teacher attempts to engage some students [_] No 


6. Does the teacher provide explicit feedback on student work and their participation in 
small-group learning activities? 


[_] Yes [_]teacher provides feedback to some students [_] No 


7. Does the teacher make explicit connections between the Small-Group learning activities 
and those included in the earlier Whole-Group session? 


[_] Yes [_] No 


Independent Reading Start time: End time 


1. Do students using the Audiobooks appear to be listening and following along with the 
text? 


[_] Yes L_] Some [_] No [_] No, because students are not using Audiobooks 
2. Are students writing in reading logs or journals? 


[_] Yes [_] Some ate writing in logs or journals |_] No 


Computer Rotation Start time: End time 


1. Do the students appear to be on task? 
[_] Yes [_ ]Some are on task [_] No 

2. Do any of the students appear to be having trouble using the computers? 
[_] Yes, some are having trouble [_] No 


If students have trouble, do they receive help quickly? 


[_] Yes [_] No 


Whole-Group Wrap-Up Start time: End time 


1. Does the teacher review key points from the lesson? 
[_] Yes [_] No 
2. Do students reflect on literacy or learning experiences? 


[_] Yes [_]No 


III. Classroom Management 


Based on the entire observation of the READ 180 class, answer the following questions. 


de 


Are expectations for rotations, student work, and behavior clear and explicit? 
[_] Yes, as indicated by clear directions from the teacher 
[_] Yes, as indicated by displays that are posted on classroom walls and elsewhere 


[_] No 


Is there disruptive behavior that interrupts the classroom instruction and student 
movement from one rotation to the next? 


[_] Yes [_]No 


IV Teacher Questions 


1. 


Were any students absent today? If so, how many students? 


Are all of the students listed in SAM? 


How often do students take the SRI? 


(ONLY ASK IF MATERIALS OTHER THAN RBOOK WERE USED IN WHOLE 
GROUP OR SMALL GROUP SESSION) I noticed that you used some materials that 
were not READ 180 in whole group/small group. Why is that? 


Was today a typical lesson? Did I observe anything that was unusual for your class? 


Appendix E 


Whole School Student Impact Table 


Table 74. Whole school impact analyses 


Fixed Effects 


Effect 
Intercept 
Male 
African-American 
Hispanic 
Special Education 
Treatment (Baseline) 
Uniform Changes, Year 1 
Treatment Effect in Year 1 
Uniform Changes, Year 2 
Treatment Effect in Year 2 
Uniform Changes Year 3 
Treatment Effect in Year 3 


Intercept 


Estimate 
231.7 
-0.096 
-0.27 
-0.16 


Standard 
Error 
10.52 

0.18 
0.98 
0.09 
0.22 
3.42 
2°01 
3.01 
2.20 
3.07 
2.14 
3.04 
10.52 


DF 
30 
85 
85 
85 
85 
85 
85 
85 
85 
85 
85 
85 
30 


t Value 
22.01 
-0.54 
-3,52 
1.68 
-1.37 
0.15 
-5.43 
-1.12 
-5.36 
-1.16 
-0.86 
-0.58 
22.01 


Pr> |t| 
<0.0001 
0.59 
0.0007 
0.09 
0.17 
0.916 
<0.001 
0.27 
<0.001 
0.25 
0.39 
0.56 
<0.0001 
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